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SUMMARY 


The  organization  of  this  report  is  intended  to  aid  reading  to  the 
level  of  detail  required  for  individual  purposes.  The  following  stages 
of  increasing  detail  are  pointed  out  to  show  the  reader  the  various 
points  at  which  he  mav  stop  if  it  suits  his  needs  for  information: 


1.  A one-page  Abstract  is  included  in  this  document. 


2.  Chapter  I of  this  report  constitutes  a short  summary  of  the 
entire  program.  An  overview  of  the  program  and  principal  results  are 
presented  which  may  suffice  for  executive-level  personnel,  or  as  an 
initial  reading  for  technical  personnel. 


3.  The  two  major  products  of  this  study  are  (1)  a definition 
of  performance  measures  appropriate  to  combat-crew  training  needs, 
and  (2)  a definition  of  a cost-effective  measurement  system  usable 
in  combat-crew  training  environments.  The  chapters  of  this  report, 
subsequent  to  Chapter  I,  amplify  on  these  study  products. 


4.  A number  of  detailed  interim  technical  reports  were  prepared 
as  working  documents  during  the  course  of  this  contract,  in  the 
pursuit  of  a number  of  areas  related  to  this  study.  Six  of  these 
are  included  as  separate  Volumes  to  this  document.  They  are: 


AFHRL-TR-74-108(TT) : Phase  I.  Measurement  Requirements 

AFHRL-TR-74-108(ITI) : Phase  IT.  Measurement  System  Requirements 

AFHRL-TR-74-108  (IV):  Phase  ITIA.  Crew  Performance  Measurement 


AFHRL-TR-74-108(V) : Phase  IIIB.  Aerial  Combat  Maneuvers 

Measurement 


AFHRL-TR-74-108 (VI) : Phase  IIIC.  Design  Studies 


AFHRL-TR-74-108(VII) : Phase  HID.  Specifications  and 

Implementation  Plan 


PREFACE 


This  is  the  Final  Report  of  Contract  F41609-71-C-9008, 
entitled,  "Research  on  Operational  Combat-Ready  Proficiency 
Measurement."  This  contract  was  performed  by  Manned  Systems 
Sciences,  Inc.,  Northridge,  California,  for  the  Flying  Training 
Division,  Air  Force  Human  Resources  Laboratory  (AFSC) , Williams 
Air  Force  Base,  Arizona.  Major  J.  A.  Fitzgerald,  Chief,  Combat- 
Crew  Training  Branch,  was  the  Contract  Monitor.  Richard  W. 
Obermayer  served  as  the  Principal  Investigator. 
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ABSTRACT 


To  improve  acquisition  of  training  performance  information  through 
usable  tools,  this  study  was  directed  to:  (1)  Systematic  definition  of 
performance  measures  appropriate  to  combat- training  needs.  (2)  Definition 
of  a cost  effective  measurement  system  usable  in  combat-crew  training 
environments  to  acquire  and  process  needed  training  information. 


Definitions  of  needed  performance  measures  were  based  on  data 
derived  from  data  collection  trips  to  Castle  AFB  (B-52) , Altus  AFB 
(C-141),  Dyess  AFB  (C-130),  Davis-Monthan  AFB  (F-4),  Tyndall  AFB 
(F-106),  Luke  AFB  (A-7),  George  AFB  (F-4  crew),  Norton  AFB  (C-141 
crew),  Nellis  AFB  (F-4  ACM).  System  criteria  were  based  on  an  analysis 
of  combat-crew  training  research  procedures.  Design  studies  were  per- 
formed to  provide  data  for  tradeoffs  between  alternative  system 
candidates.  Finally,  detailed  system  specifications  and  an  implemen- 
tation plan  were  prepared. 


Performance  Measures 


Measurement  was  generated  in  the  following  steps:  (1)  The  require- 
ments of  six  aircraft  were  consolidated  into  a common  framework  of 
maneuvers:  Transition,  Instruments,  Formetion,  Air  Combat,  Air 
Refueling,  Ground  Attack,  Air  Drop,  Radar  Navigation  and  Bombing; 

(2)  For  each  maneuver,  data  collected  from  CCTS  visits  were  formalized 
into  measurement  requirements;  (3)  Measurement  parameters  and  (4) 
specifications  were  produced  for  hardware  and  software  implications, 
respectively;  and  (5)  Examination  of  crew  interactions  led  to  an 
analysis  of  communications  measurement. 


The  Performance  Measurement  System 


The  performance  measurement  system  consists  of  the  following 
subsystems:  (1)  Data  Acquisition.  A hybrid  audio/video/photo/digital 
recording  system  with  programmed  recording  control  was  derived  from 
system  tradeoff  analyses.  A data  playback  station  for  combined 
manual  and  automatic  processing  is  provided.  (2)  Data  Processing. 

A general  purpope  digital  computer  with  standard  peripherals  is 
required.  In  addition  to  executive  and  utility  programs,  specialized 
input,  edit,  measurement  and  analysis  software  is  needed.  (3) 

Personnel.  A crew  consisting  of  the  following  types  (1)  system 
director,  (2)  research  personnel,  (3)  computer  programmer,  data  clerks, 
engineers  and  technicians,  and  secretary,  is  needed  to  perform 
functions  leading  to  research  objectives.  (4)  Facilities.  Two 
mobile-home  trailers  are  recommended  to  house  personnel  and 
ground-based  equipment. 
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INTRODUCTION  AND  SUMMARY 


Today,  assessment  of  aircrew  proficiency  in  those  skills 
associated  with  advanced  flying  training  depends  largely  on 
subjective  evaluations  by  instructor  pilots,  supplemented  by  the 
more  or  less  objective  gunnery  and  bombing  scores.  In  some 
areas,  such  as  air  combat  maneuvering,  combat-readiness  determi- 
nation is  purely  subjective,  wholly  without  even  the  meager 
objective  material  available  in  other  areas.  An  economically 
acceptable  means  of  objectively  measuring  behavioral  skills  in 
the  operational  or  crew  training  setting  has  been  an  elusive 
goal . 


Interest  in  the  problem  has  intensified  in  recent  years 
because  of  the  projected  increase  in  costs  of  operating  the  newer 
weapons  systems  over  the  present  generation  of  combat  aircraft. 
Aircrew  training  costs  currently  absorb  one  quarter  of  the  Air 
Force's  budget  and  has,  therefore,  become  an  area  of  fiscal 


concern . 


The  key  to  significant  cost  savings  lies 
directed  to  the  reduction  of  flying  hours  duri 
the  transfer,  where  possible,  of  such  training 
devices.  Illustrative  of  the  magnitude  of  the 
is  a letter  from  the  Commander  of  the  Tactical 
CSAF 1 , in  which  the  TAC  Commander  describes  an 
aircrew  training  class  in  which  a reduction  of 
per  student  would  result  in  an  estimated  $700, 


in  an  approach 
ng  training,  and 
to  lower  cost 
possible  savings 
Air  Command  to  the 
experimental 
35  flying  hours 
000  cost  savings. 


The  USAF  is  attempting  to  adapt  modern  Systems  Approach  to 
Training  (SAT)  techniq -es  to  aircrew  training  programs2.  Inherent 
in  the  SAT  concept  is  student  advancement  on  individual  profi- 
ciency rather  than  course  length.  For  the  more  complex  aircrew 
training  objectives,  the  traditional  subjective  evaluative 
methods  may  prove  insufficient. 


Objectives 


In  an  effort  to  improve  training  performance  information, 
this  study  was  directed  to  two  objectives: 


Goal  1.  Systematic  definition  of  performance  measures 
appropriate  to  combat-crew  training  needs. 
Performance  measures  will  include  formal 
statements  of  methods  of  measuring  flight 


'Ltr,  CC  TAC  to  CSAF,  14  Mar  70,  Subject:  "Flying  Training 
Efficiency . " 


2Ltr,  CSAF  to  MAJCOMS , 2 Feb  70,  Subject:  "Flying  Training 
Efficiency." 
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crew  performance  used  during  and  at  the  end 
of  combat-crew  training,  and,  new  measures 
meaningful  to  combat-crew  training  and 
useful  as  tools  for  training  research. 

Goal  2.  Definition  of  a cost-effective  measurement 
svstem,  usable  in  combat-crew  training 
environments  to  acquire  and  process  needed 
training  information.  Specification  of  the 
recommended  system  includes  all  data 
acquisition  devices  and  methods,  data 
processing  hardware  and  software,  facilities, 
and  personnel  necessary  for  training  research. 

This  study  thus  was  an  effort  to  describe  usable  measurement 
tools  for  utilization  in  combat-crew  training  research. 

Instructional  system  development.  Research  studies  directed 
toward  performance  measurement  in  combat-crew  training  are  highly 
relevant  today  in  view  of  USAF  policy  to  employ  a systems 
approach  to  flying  tiaining  problems.  The  model  for  Instructional 
System  Development  (cf.,  Dept,  of  Air  Force,  1970)  contains  the 
following  basic  steps: 

1.  Analyze  system  requirements. 

2.  Define  education  or  training  requirements. 

3.  Develop  objectives  and  tests. 

4.  Plan,  develop,  and  validate  instruction. 

5.  Conduct  and  evaluate  instruction. 

In  support  of  instructional  system  development,  measures  and  a 
measurement  system  are  necessary  to  (1)  perform  analyses  of 
systems  in  their  operational  environments,  (2)  establish 
quantitative  instructional  standards,  (3)  provide  an  index  of 
achievement  for  each  behavioral  objective,  and  (4)  evaluate 
alternative  instructional  content,  approaches,  and  training 
devices . 

In  particular,  instructional  system  development  requires 
that  performance  standards  are  identified  so  that  the  most 
efficient  approach  is  used  to  train  the  needed  skills  and 
knowledge  to  the  dvired  level  of  performance.  Such  performance 
standards  imply  pr  ' rmance  measurement  for  both  the  determination 
of  desirable  appr  . a0,  to  training  and  for  testing  student 
performance. 

A measurement  system  for  the  operational  environment.  It 
was  required  to  develop  measurement  tools  that  would  be  usable  in 
the  operational  environment  under  the  constraints  that  such  an 
environment  implies.  Within  the  context  of  this  requirement,  it 
was  necessary  to  establish  a list  of  parameters  to  be  sensed,  and 
the  point-of-view  taken  was  that  the  parameters  should  be  derived 
from  that  information  that  the  operational  training  personnel 
consider  to  be  meaningful  and  significant. 
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An  automated  measurement  system.  An  ancillary  objective  was 
to  develop  a measurement  system  that  would  relieve  the  instructor 
pilot,  to  a maximum  extent,  from  the  requirement  of  having  to 
record  a great  deal  of  information  manually  on  the  basis  that 
such  activity  degrades  his  ability  to  competently  instruct  his 
student.  This  does  not  imply  that  such  a measurement  system  is 
an  attempt  at  automated  evaluation.  The  measurement  system 
should  certainly  include  means  for  transforming  and  analyzing 
performance  information,  but  ultimately  evaluation  and  decision 
for  training  control  is  a human  function. 


Measurement  Based  on  Combat-Crew  Training  Information  Needs 


The  strategy  employed  was  to  design  a measurement  system 
that  could  acquire  that  data  identified  as  meaningful  by  training 
management  and  instructor  personnel.  On-site  visits  were  made 
across  a broad  spectrum  cf  combat-crew  training  programs  to  define 
these  data. 


The  combat-crew  training  sites  visited  are  listed  in  Table  1. 
The  aircraft  sample  included  heavy  (inter-  and  intra-theatre 
cargo/transport,  and  bomber)'  and  high-performance  aircraft  (one- 
and  two-man  interceptor  and  tactical  fighter) . An  attempt  was  made 
to  (1)  consider  measurement  in  the  context  of  combat-crew 
training,  (2)  assess  measurement  already  included  as  well  as 
identify  potential  measurement  indicated  by  combat-crew  training 
personnel,  and  (3)  assess  the  constraints  placed  by  the 
environment  on  feasible,  usable  measurement  systems. 


Measurement  System  Criteria 


While  definition  of  measurement  was  based  on  training 
information  needs,  the  criteria  for  the  measurement  system  were 
based  primarily  on  consideration  of  the  use  of  measurement  to 
achieve  research  goals.  To  achieve  research  goals,  measurement 
must  produce  information  in  a timely  and  useful  form  during 
typical  research  activities.* 


Typical  research  sequence.  A typical  research  sequence  is 
shown  In  Figure  1. 


(1)  The  test  plan  and  measurement  specification  provides  an 
initial  statement  of  research  crew  activities  in  achieving 
specific  data  answering  to  the  research  problem. 


(2)  For  the  data  to  lead  to  meaningful  measurement,  the 
research  crew  must  be  very  familiar  with  expected  performance 
and  difficulties  which  will  be  met;  consequently,  they  must  be 
aware  of  the  goals  of  each  mission  as  planned  and  accomplished, 
and  all  factors  related  to  performance  on  each  flight  (mission 
briefing/debriefing  and  flight  monitoring) . 
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TABLE  1 

DATA  COLLECTION  SITES 


PLACE 

AIRCRAFT 

Castle  AFB 

B-52  F,  G & H 

Altus  AFB 

C-141A 

Dyess  AFB 

C-130E 

Davis-Monthan  AFB 

F-4  C,  D & E 

Tyndall  AFB 

F-106  A & B 

Luke  AFB 

A-7D 

♦George  AFB 

F-4E 

♦Norton  AFB 

C-141A 

♦♦Nellis  AFB 

F-4E 

li 


♦Special  emphasis  on  crew-performance  measurement. 

♦♦Special  emphasis  on  air-combat  maneuvers 
performance  measurement. 
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TEST  PLAN  I BRIEFING 


Figure  1.  Typical  Research  Sequence. 


(3)  Data  processing  is  ordinarily  an  iterative  procedure, 
trying  alternatives  and  adapting  to  performance  characteristics, 
until  the  best  data  forms  are  achieved. 

(4)  All  known  data  collection  systems  permit  frequent 
errors,  making  data  editing  a major  and  important  task  which 
requires  digital  computer  support. 

(5)  Data  analysis,  like  measurement  development,  is  an 
iterative  process,  searching  for  the  best  means  to  fully  present 
experimental  results;  analytical  tools  involve  large  libraries 
of  computer  programs  for  statistical  treatment. 

(6)  In  general,  the  research  crew  will  be  responsible  for 
using  the  facility  described  to  produce  a technical  report 
answering  to  specific  research  objectives. 

System  criteria.  The  following  criteria  were  used  for 
tradeoff  analysis  of  candidate  measurement  systems  and  to  guide 
design  of  the  selected  approach. 

(1)  Provide  needed  information.  If  a measurement  system  is 
viewed  as  an  information  system  it  becomes  apparent  that  the 
most  important  consideration  is  that  it  produce  the  information 
needed,  i.e.,  primarily  the  information  which  was  identified 
with  the  aid  of  combat-crew  training  personnel. 

(2)  Provide  data  in  a useful  form.  Data  should  be  provided 
in  a useful  form  for  (l)  research  objectives,  and  (2)  combat- 
crew  training. 

(3)  Short  research  cycle  time.  So  that  measurement  is 
sufficiently  timely,  the  measurement  system  must  be  pet-up  for 
operation  as  soon  as  possible  after  a research  topic  is 
identified,  the  system  must  be  reliable  to  allow  data  collection 
to  keep  pace  with  training  schedules,  and,  processed  data  must 
be  available  shortly  after  each  flight  to  permit  effective 
review  (within  one  hour  for  debriefing  purposes,  within  12  hours 
for  research  review) . 

(4)  Minimum  initial/sustaining  costs.  Initial  costs  include 
modification  of  devices,  installation,  design  and  equipment 
purchases.  Sustaining  costs  include  personnel  salaries,  facility 
leases,  supplies  and  services.  These  costs  must  be  traded-off 
against  the  utility  of  the  information  derived. 

(5)  Minimum  data  distortion.  Data  distortion  must  be 
controlled  so  that  missing  and  uninterpretable  data  do  not  render 
research  meaningless. 

(6)  Compatible  with  different  training  devices.  The 
measurement  system  should  be  able  to  support  training  and 
research  which  may  take  place  with  a wide  range  of  existing  or 
proposed  training  devices . 


(7)  Permit  iterative  design  of  valid  measurement.  Itera- 
tions in  measurement  design  must  be  sufficiently  convenient  so 
that  valid  measurement  appropriate  to  research  needs  may  soon 
evolve . 
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(8)  Minimum  interference  with  training  processes,  A mea- 
surement system  for  training  research  should  be  inconspicuous, 
requiring  little  or  no  attention  from  the  student  or  instructor. 

(9)  Permit  correlation  with  data  from  external  sources.  For 
any  given  study,  data  from  a number  of  such  data  sources  may  be 
collected,  combined  and  correlated. 

(10)  Minimum  space,  weight,  cooling  and  power  requirements. 
Data  acquisition  equipment  must  be  installable  in  aircraft  of the 
present  and  future  U.S.  Air  Force  inventory,  including  fighter- 
type  aircraft. 

(11)  Effective  self-sufficient  personnel/facility  configura- 
tion. A proper  mix  of  people,  tools,  supplies,  faclli ties,  and 
support  are  needed  to  successfully  sustain  and  use  a performance 
measurement  system  for  training  needs. 

I 

Product  1,  Combat-Craw  Performance  Measurement 
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To  a large  extent,  a common  basis  for  measurement  was 
established  for  the  six  aircraft  included  in  the  study;  thus, 
allowing  for  a more  or  less  modular  approach.  Allowance  must  be 
made  for  special  aircraft  characteristics;  for  example,  the 
F-106  has  no  flaps  and  the  B-52  has  a quite  complicated  flap 
retraction  routine  compared  to  other  aircraft.  Measurement  was 
treated  for  each  of  the  following  maneuvers: 
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(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

(8) 
(9) 

(10) 


Takeoff  and  Climb 

Pattern,  Land  and  Go-Around 

Instruments — General  and  Example 

Formation 

Air-Air  Intercept 

Air  Combat  Maneuvers 

Air  Refueling 

Ground  Attack 

Air  Drop  and  Air  Drop  Formation 
Radar  Navigation  and  Bombing. 


Prototype  measurement.  For  each  of  these  maneuvers,  the 
data  collected  from  combat-crew  training  site  visits  were  compiled 
into  the  summary  form  shown  in  Figure  2.  Since  the  blanks  in  the 
summary  form  of  Figure  2 indicate  needed  information,  and  conse- 
quently items  for  measurement  development,  these  forms  were 
termed  Prototype  Measurement  since  they  form  a model  after  which 
measurement  could  be  patterned. 


f A&rt  Vila*  Jfc.  itiv ai' ^ ^?I ^ Vfr 


TAKEOFF  & CLIMB* 


CONDITIONS: 


Gross  Wt: 
Temp . : 


Wing: 

Alt. Set. : 


/ Runway: 
Field  Elev. 


_/ 

Form  Pos . : 


TAKEOFF  ROLL:  (TO  power  until  rotation) 

Power  Set:  Centerline  Dev.:  Min,  Max,  Av. 

Reject  Speed:  Computed  Heading:  Min,  Max,  Av. 

Time:  Dist:  Bank:  R Max,  L Max 

ROTATION:  (Nose  gear  off  until  pitch  att.  established) 


Rot.  Speed:  

Pitch:  Rate: 

Final: 


Stab.  Trim: 
Bank: 


Overhsoot: 


Centerline  Dev. : 
Heading:  


LIFTOFF : (Pos.  Vert.  Vel.) 


Unstick  Speed:  

Vert.  Vel.  After: 


Pitch: 


Bank: 


Hdg: 


Sec.  : 


GEAR-UP : (Handle  up  until  gear-up  & locked) 

V.V.  Init.:  V.V.  Final: 

Hdg:  


Gear-Up  Speed: 
Pitch:  Bank: 


FLAPS  UP:  (Start  up  to  full  up)  Note:  F106  has  no  flaps 


Trim:  

B-52  Only 

IAS 

PITCH 

ALT 

w 

TRIM 

Pitch: 

Bank:  Hdg: 

- ~ 

X 

V 

A/S  (INIT) 

(FINAL) 

Dlall 

1st  Pos 
2nd  Pos 
Full 

X 

X 

v 

X 

X 

V 

W (INIT) 

(FINAL) 

X 

X 

X 

X 

X 

v 

X 

V 

X 

v 

ALT  (INIT) 

(FINAL) 

X 

X 

X 

X 

X 

X 

X 

X 

> 


y 

| 

« 


CLIMB  & LEVEL-OFF : (Depends  on  Flight  Plan) 

INIT  FINAL 


j 

V 

PWR 

A/S  MACH 

HDG 

ALT 

ALT 

PITCH  TRI1 

r 

Accelerate 

X 

X 

X 

X 

X 

X 

X 

X 

t j 

l 

j? 

Clir’->  A/S 

(#1) 

(#2) 

X 

X 

X 

X 

X 

X 

X 

X 

i 

Climb  MACH 

X 

X 

X 

X 

X 

X 

X 

X 

l 

pi 

Level-Off 

(Alt-10%  W) 
(to  Cruise) 

X 

X 

X 

X 

X 

X 

X 

X 

♦Also,  mandatory  communication  & instances  where  A/C  limits  are 
exceeded. 


Figure  2.  Example  of  Prototype  Measurement. 
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parameters)  can  be  determined. 


For  example,  Figure  2 Indicates  that  a measure 

deviations  is  desired  during  ^“*g°“or.oi'  the  runway  and  the 
the  distance  between  the  ‘i^^P^ISr Lavement.  The 
runway  centerline  is  a paranh.  , average  difference,  or, 

desired  measurement  might  be  ^®shi  ^tween  centerline 

conceivably,  might  involve  th  brake  application)  and 

deviation  and  heading  (or  “alters  to  execute  the 

thereby  indicate  the  nee  Further  the  measurement  calcula- 

SSl atf  ^ure  2,  From  the  application 

parameters  TOSSS  ^e'm^Snflnterval  starts  and 
stops 


..  j_  a^rfttion  to  basic  obvious  parameters,  the 

following6 types^of  £22£  m£  be Jor^utation,^ 

(1)  parameters  for  impiementing  ogi«  ,ed  to  deslred  performance, 
anf  (irerwr  infir^alion  derived  fr«  the  difference  between 
actual  and  desired  performance r 


Therefore,  “^^thePrototypeMeasurement  «^rJ^*JaU 

processing  U 

sur^ss  r^rnSti-Jt  ir.  «rsr 

scribe^dat^cler^procedure^where  manual  processing  rs  employed. 


Product  2.  Performance  Measurement  System 


=Ss$aiSigSKt 

inf ormationWrequirements . ^ybrid  system^as  ^refore^ecom- 

system*  woul^not*  b^neededforall  ^"^^SfLr^pend- 

ssrs  g.-sai*sd=ES  «Sr^y^ 

the^igital^recording  S*S&2T  Recording  is  provided 
with  either  modular  subsystem. 


„i2rs  ms  rris^ssra 

presented  in  Chapter  III  of  this  report. 
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Figure  3.  Example  Measurement  Specification 


AIRCRAFT/SIMULATOR 
DATA  COLLECTION 


Audio-Video 

Recorder 


Dual  Camera 


.Programmed 

Recording 

Control 


Discrete  Signal  Recording 

Audio-Digital 

Recorder 

16+  Analog  Channels 
24  Discrete  Channels 
Digital 

Auxiliary  Camera 


Mobr'  *.e 

Test 

Stand 


FIELD  DATA  CC  LECTION 


BRIEFING/DEBRIEFING 


Data 

Acquisi- 

tion 


Instrumentation 
Camera  with 
Clock  Inset 
Intervalometer 
Audio  Recorder 
Transceivers 


Small  Camera 
and 

Cassette  Recorder 


EXTERNAL  DATA  COLLECTION! 


Data  in  Digital 
Format 
Cards 
Mag  Tape 
Paper  Tape 


DOCUMENTARY  DATA 
COLLECTION 


Manual 

Digital  Entry 
Equipment 


1 

Data 

Playback 

and 

\ 

l 

V 

Processing 

£ 

i 

j 

Personnel 

m 

and 

/S?;, 

H*!? 

Facilities 

n 4 J 

Offices  General  Purpose  Video  Playback 

Debriefina  Computer  Photo  Viewers 

Tapa/Disc/Card  Audio  Playback 
Teletype/Line  Printer/Plotter  Playback  Displays 
Storage  Conversion  Equip.  & Controls 


GROUND  DATA  PROCESSING 


Research  Scientists  Programmer 

Data  Clerks  Engineers/Technicians 

Secretary 


Single  or  Dual  Trailer 
Conf i guratlon 


Figure  4.  Mobile  Combat-Crew  Performance  Measurement  System. 
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Data  acquisition.  Data  acquisition  must  provide  for  the 
following  data  sources;  (1)  aircraft  and  simulator,  (2)  field 
(e.g.,  runway  or  range),  (3)  briefing  and  debriefing,  (4)  exter- 
nal sources  (e.g.,  other  studies),  and  (5)  documents.  The  major 
amount  of  data  acquisition  equipment  is  required  for  aircraft/ 
simulator  performance  data.  A hybrid  audio/video/digital  record- 
ing system  was  specified,  including  semi-automatic  programmed 
recording  control.  An  auxiliary  camera  is  required  for  some 
mission  types  (e.g.,  air  drop)  and  provided  as  a supplement  for 
others.  To  reduce  costs  the  digital  recording  capability  was 
restricted,  but  sufficient  capacity  exists  to  provide  stand-alone 
digital  recording  for  many  applications.  The  video  recording 
equipment,  which  is  the  source  of  all  pictorial  or  out-the-window 
information,  is  augmented  with  discrete  recording  channels  (for 
events  such  as  wheels-up,  speed  brakes  out,  weapon  firing,  etc.) 
to  expand  video  recording  capacity  and  permit  rapid  manual 
processing  (by  providing  convenient  stopping  points  used  to  key 
manual  data  reading) . 


Data  playback  and  processing.  It  is  clear  that  audio  and 
video  data  recording  requires  manual  processing,  while  digital 
recording  permits  use  of  a high  degree  of  automation.  Data 
processing  must  therefore  provide  for  integrated  manual  and 
automated  data  playback.  A general-purpose  digital  computer  with 
many  standard  peripherals  is  needed  to  (1)  input  data,  (2)  edit 
data  (man-computer  interaction),  (3)  compute  measures,  and 
(4)  perform  data  analyses.  Standard  executive  software  and  user 
languages  (FORTRAN  IV)  must  be  augmented  with  special  measurement 
processing  programs. 


Personnel  and  facilities.  The  system  is  housed  in  either  a 
single-'  or  dual-trailer  configuration,  employing  a structure 
used  fob  mobile  homes.  In  this  fashion,  the  ground-based  portion 
of  the  system  is  easily  transported  to  a new  data  collection 
site  and  rapidly  readied  for  use.  Example  layouts  are  shown  in 
Figure  5. 


An  integrated  research  team  capable  of  all  research  functions, 
starting  with  problem  identification  and  following  through  with  a 
technical  report,  is  recommended.  The  personnel  complement  will 
include  (1)  Research  Scientists,  (2)  Programmer,  (3)  Data  Clerks, 

(4)  Engineers  and  Technicians,  and  (5)  Secretary.  The  crew 
conducting  the  research  must  possess  a blend  of  skills  and 
knowledge,  including  the  following  experience:  (1)  U.S.  Air 

Force  Combat  flying,  (2)  training  psychology,  (3)  experimental 
design,  (4)  field  research,  (5)  statistical  analyses,  and 
(6)  programming. 


More  Detailed  Information 


The  remainder  of  this  report  expands  upon  the  two  principal 
study  products:  Chapter  II.  Combat-Crew  Performance  Measurement, 

and,  Chapter  III.  A Mobile  Combat-Crew  Performance  Measurement 
System.  Chapter  IV  discusses  implications  for  development  of  the 
system  described. 
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Figure  5.  Example  Mobile  Measurement  System  Configuration  (12'  x 60'  Trailers). 


For  further  detail  on  the  principal  study  products,  as  well  as  the 
analyses  and  methods  used,  reference  should  be  made  to: 


AFHRL-TR- 74-108(11) : Phase  I.  Measurement  Requirements 

AFHRL-TR-74-108(III) : Phase  II.  Measurement  System  Requirements 

AFHRL-TR- 74-108 ( IV) : Phase  IIIA.  Crew  Performance  Measurement 


AFHRL-TR- 74-108 (V) : Phase  IIIB.  Aerial  Combat  Maneuvers  Measurement 

(Secret) 


AFHRL-TR- 74- 10 8 (VI) : Phase  IIIC.  Design  Studies 


AFHRL-TR- 74- 10 8 (VI I) : Phase  HID.  Specifications  and  Implementation 

Plan 
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II.  DERIVATION  OF  COMBAT-CREW  PERFORMANCE  MEASUREMENT 

! 

One  cf  the  principal  products  of  this  study  was  the 
definition  of  performance  measures  appropriate  to  combat-crew 
training  needs.  Based  on  interviews  with  instructors  and 
training  management,  measurement  definitions  evolved  in  the 
following  steps: 

(1)  The  varied  requirements  posed  by  the  six  aircraft  and 
missions  were  consolidated  into  a common  framework  which 
permitted  isolation  of  measurement  modules  (measurement 
commonality)  ; 

(2)  Discussions  of  operational  training  information  needs 
were  formalized  to  indicate  in  a checklist  fashion  the 
measurement  development  needed  (prototype  measurement) ; 

(3)  Measurement  parameters  and  (4)  measurement  specifica- 
tions were  produced  together,  but  are  presented  separately  for 
hardware  and  software  implications,  respectively; 

(5)  Measurement  and  analysis  for  crew  communicatio.i ; 
recording  were  examined  to  provide  means  of  examining  crew  inter- 
actions and  individual  contributions  to  total  man-machine  system 
performance. 

Measurement  Commonality 

Common  measurement  was  desired  to  permit  the  desion  of  one 
simple  and  practical  measurement  system,  and  eliminate  the  need 
for  a totally  unique  measurement  system  for  each  aircraft. 

Common  training  phases.  As  a first  step  to  assessing 
com.nonality  of  measurement  requirements,  similar  training  phases 
were  compared  for  each  aircraft.  Table  2 presents  the  training 
phases  compared. 

Not  all  maneuvers  were  taught  at  all  sites  visited.  For 
e ample,  tbs.  operational  C-130  squadron  visited  did  not  explicitly 
train  transition  maneuvers;  nevertheless,  competent  information 
was  obtained  for  measurement.  The  F-106  air  refueling  modifica- 
tion had  not  been  completed,  and  therefore  was  not  trained.  All 
combat  maneuvers  were  not  taught  at  the  combat-crew  training 
squadrons  visited,  but  attempts  were  made  to  fill  these  gaps  in 
data  collection  by  cross-checking  with  other  aircraft  training 
wftere  possible. 

The  following  observations  are  indicative  of  the  degree  of 
commonality  of  training  phases  for  the  six  aircraft: 

(1)  Transition  (TR) . Transition  maneuvers  occur  in  a 
common  fashion,  but  the  manner  in  which  they  are  performed 
varies. 


IMiLsiSM 
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TABLE  2 

COMMON  TRAINING  PHASES 


MULTI  HEAVY 

HI  PERF 

B-52 

C-141 

C-130 

F-106 

F-4 

A- 7 

TR 

TR 

TR 

TR 

TR 

TR 

INST 

INST 

INST 

INST 

INST 

INST 

- 

FORM 

FORM 

FORM 

FORM 

FORM/ BFM1 

AA 

AA 

3FM/ACM 

FORM/DFM 1 

AR 

- 

AR 

AR 

AIR  DROP 

AIR  DROP 

GA 

GA 

HI 

LOW 

(BOMB) 

BEACON 

DROP 

GAR 

RNB 

LEGEND 


TR: 
INST: 
FORM: 
BFM: 
AA: 
ACM: 
AR: 
GA: 
GAR: 
RNB: 


Transition 

Instruments 

Formation 

Basic  Flight  Maneuvers 

Air-Air  Intercept 

Air  Combat  Maneuvers 

Air  Refueling 

Ground  Attack 

Ground  Attack  Radar 

Radar  Navigation  Bombing 

Data  Not  Available  at  Sites  Visited 


formation  and  Basic  Flight  Maneuvers  are  combined  into  one 
phase  for  A-7D  training;  therefore,  a portion  is  similar 
to  formation  training  on  other  aircraft,  and  a portion  is 
similar  to  air-combat  maneuvers  training  for  the  F-4 
aircraft. 


(2)  Instruments  (INST) . Instrument  maneuvers,  and  the 
external  criteria  which  must  be  met,  are  essentially  the  same. 


(3)  Formation  (FORM) . Formation  flight  is  performed  by  all 
six  aircraft  as  a means  to  optimally  employ  composite  flight  and 
provide  individual-ship  effectiveness.  However,  a number  of  types 
of  formation  exist  for  specialized  missions,  each  appropriate 
only  to  specific  aircraft. 


(4)  Air-Air  Intercept  (AA) . Air-to-air  intercept  and 
weapons  delivery  utilizing  airborne  radar  is  accomplished  with 
only  the  F-4  and  F-106  aircraft  (rendezvous  for  air  refueling  is 
a different  problem) . While  the  F-4  and  F-106  maneuvers  and 
equipment  differ,  the  same  basic  measurement  requirements  are 
presented. 


(5)  Basic  Flight  Maneuvers  (BFM)  and  Air  Combat  Maneuvers 
(ACM) . BFM  and  ACM  are  grouped  together  in  F-4  training,  while 
BFM  and  Formation  are  grouped  together  in  A- 7 training.  The 
BFbl/ACM  grouping  was  maintained  for  purposes  of  measurement 
development,  but  formation  was  treated  as  a separate  measurement 
problem. 


(6)  Air  Refueling  (AR) . Air  refueling  can  occur  with  four 
of  the  six  aircraft,  but  is  only  considered  a difficult  maneuver 
for  the  B-52.  Consequently,  air  refueling  for  the  B-52  was 
emphasized. 


(7)  Ground  Attack  (GA)  and  Air  Drop.  A number  of  training 
phases  are  devoted  to  F-4  and  A- 7 ground  attack  (Day,  Tactical, 
Night,  with  various  weapons  and  delivery  modes) , but  common 
measurement  was  judged  to  be  appropriate. 


Air  Drop  (Combat  Airlift  Mission)  maneuvers  for  C-141 
and  C-130  training  perhaps  superficially  resemble  level-flight 
ground  attack,  but  quite  different  measurement  requirements  are 
posed. 


(8)  Radar  Navigation  and  Bombing  (RNB) . Navigation  by  use 
of  radar,  and  subsequent  delivery  on  target  of  either  ordinance 
or  cargo,  occurs  with  most  of  the  aircraft  of  the  sample  except 
the  F-106,  and  compatible  mission  performance  measurement  is 
indicated. 


Composite  of  training  phases.  From  this  examination,  it  is 
goncluded  that  measurement  can  be  directed  to  a composite  of  the 
training  phases  for  the  sample  of  six  aircraft,  as  follows: 


Transition 
Instruments 
Formation 
Air-Air  Intercept 
Air  Combat 


4*0: 
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Air  Refueling 
Ground  Attack 
Air  Drop 

Radar  Navigation  and  Bombing 


Of  course,  all  of  these  training  phases  are  not  applicable  to  all 
aircraft  of  the  sample,  requiring  only  that  the  measurement 
subsequently  developed  be  ignored  where  inappropriate.  While 
there  are  differences  in  measurement  requirements  for  specific 
aircraft  within  a given  "common”  training  phase,  these  are  slight 
compared  to  measurement  differences  across  training  phases. 


Common  Measurement 


Each  phase  of  flight,  tentatively  considered  to  permit 
common  measurement,  was  examined  for  detailed  measurement  require- 
ments. For  each  maneuver,  measurement  requirements  were  extracted 
from  (1)  interview  notes  with  Instructors  and  Training  Managers, 

(2)  Aircraft  Technical  Orders,  (3)  Phase  Manuals,  (4)  Instructor 
Guides,  and  (5)  other  specialized  documents.  In  particular, 
important  parameters,  judging  factors  and  common  errors  were 
noted  for  each  maneuver.  The  information  which  an  instructor 
pilot,  or  training  manager,  would  consider  important  was 
especially  stressed  for  translation  into  objective  measurement. 


For  example,  measurement  requirements  were  noted  for  each 
aircraft  during  takeoff  and  climb-out  maneuvers  (Runway  Roll, 
Rotation,  Liftoff,  Gear-Up,  Flaps-Up,  Climb  and  Level-Off) . A 
matrix  of  measurement  requirements  was  thus  produced,  allowing 
comparison  across  aircraft.  It  was  noted  that  takeoff  and  climb- 
out  are  essentially  the  same  for  all  aircraft,  with  the  exception 
of  the  importance  of  some  maneuvers  (e.g.,  rotation  is  critical 
with  the  F-106) , variations  in  aircraft  design,  (e.g.,  F-106  has 
no  flaps,  B-52  involves  a complicated  flap  schedule),  and  climb 
profiles  depend  on  the  specific  mission  and  clearance. 

While  the  similarities  far  outweighed  the  differences  when 
comparing'  measurement  requirements  across  aircraft,  it  was  clear 
that  a number  of  differences  nevertheless  did  exist.  On  the 
other  hand,  it  was  apparent  that  the  basic  measurement  components 
were  the  same,  and  a small  number  of  specialized  measures  suf- 
ficed for  unique  aircraft  requirements.  In  short,  measurement 
could  be  derived  from  a relatively  small  number  of  building 
blocks  for  any  aircraft  and  maneuver.  Therefore,  a modular 
approach  to  measurement  was  justified. 


Measurement 


As  a logical  extension  of  the  considerations  of  measurement 
commonality,  examples  of  the  information  required  for  training 
were  developed  in  the  form  of  formatted  measurement  outputs. 

That  is,  the  information  needs  were  expressed  as  example  measure- 
ment, with  blanks  to  indicate  specific  measures  to  be  developed. 


These  forms  were  termed  prototype  measurement  since  they  found 
a model  after  which  measurement  could  be  patterned.  A full 
discussion  of  these  prototype  measures  is  presented  in  Volume  II. 


An  example  of  prototype  measurement  is  displayed  in  Figure  2 
for  takeoff  and  climb.  Similar  measurement  was  developed  for  each 
training  phase;  these  are  presented  in  Volume  II.  A blank  or  x 
in  Figure  2 indicates  one  or  more  numerical  entries  to  be  deter- 
mined as  a result  of  measurement.  Checking  or  mandatory 
communications,  and  other  critical  communications,  is  assumed  in 
addition  to  other  measures  of  performance  in  each  of  the 
prototype  measurement  forms. 


Cl).  Transition.  For  convenience,  transition  was  divided 
into  C1L  Takeoff  and  Climb,  and  (2)  Pattern,  Land  or  Go-around. 
As  previously  noted,  measurement  appropriate  to  the  composite  of 
six  aircraft  is  indicated;  the  F-106  and  B-52  require  special 
treatment. 


C2)  Instruments . Measurement  for  instrument  flight  is 
treated  as  the  sum  of  (1)  basic  aircraft  control  performance, 
and  (2)  navigation  performance  with  respect  to  air  traffic 
control  requirements.  While  required  measurement  modules  for 
each  of  these  classes  of  measures  can  be  specified,  it  is 
difficult  to  present  detailed  measurement  except  for  specific 
clearances  and  published  procedures. 


C3)  Formation . Measurement  for  formation  flight  is  super- 
imposed onto  mission  performance  measurement,  thus  making 
formation  measurement  difficult  to  isolate.  In  particular, 
tactical  formation  performance  was  not  clearly  identified  in 
this  study  in  an  objective  quantitative  fashion.  It  may  be 
necessary  to  rely  heavily  on  instructor  subjective  measurement 
(which  may  be  quite  satisfactory  when  the  instructor  is  in  a 
position  to  observe  performance) , except  for  measurement 
associated  with  join-up,  close  formation,  and  in-trail  formation. 


(4)  Intercept . In  order  to  be  specific,  intercept  measure- 
ment was  based  primarily  on  the  F-106,  however,  the  intercept 
problem  is  essentially  the  same  as  the  F-4.  Of  course,  a two-man 
crew  performs  in  the  F-4  (crew  performance  measurement  is 
discussed  later) . 


(51  Air  Combat  Maneuvers.  Prototype  air  combat  maneuvering 
measurement  dealt  primarily  with  set-ups  during  initial  training 
and  for  dart  firing.  A more  extensive  treatment  is  presented  in 
Volume  V.  Measurement  for  more  extensive  air  combat  training  is  indicated 
in  Table  3. 


(61  Air  Refueling.  As  Air  Refueling  is  especially  difficult 
in  B-52  combat-crew  training,  the  prototype  measurement  was 
tailored  to  the  B-52  tasks  and  to  Strategic  Air  Command  require- 
ments . 


TABLE  3 


CANDIDATE  ACM  MEASURES 


BASIC  AERODYNAMICS 


MEASURES 


DIHEDRAL  EFFECT 
ADVERSE  YAW 
LIFT  AND  DRAG 


AOA,  A/S,  ALT,  PITCH, 
ROLL,  YAW/SIDE  SLIP, 
RUDDER,  STICK. 


TURNING  PERFORMANCE 


MEASURES 


HORIZ.  G„  = tan  $ 
& VERT. 


GR2  - 1 
GT  = Gr  + COS  0 

RADIUS  & RATE  R = V2/G„_ 

OFJTURN  „ = GRg/f 

R = V2/g  (G_  - Cos  6) 


ROLL 

G 

VELOCITY  (ATTACKER) 
VELOCITY  (DEFENDER) 
ANGLE  OFF 

DISTANCE  BETWEEN  ACRFT 


PURSUIT  w = V (Sin  (AO)/S) 

CURVE 

Cos  (AO  of  Max  G)  = Vft/2V 


ENERGY  MANEUVERABILITY 


MEASURES 


ALT,  A/S,  M, 

FUEL,  THROTTLE, 

G,  W,  AOA  or  CL, 
TURN  RATE  & RADIUS. 


MANEUVERING  SEQT^NCES 

OFFENSIVE/COUNTE  -OFFENSIVE 

HI/LO  PERF 

RANGE 


MEASURES 

ENERGY/AERO  COND. , 
FLIGHT  PATH,  VEH.  ATT., 
THRUST,  ANGLE-OFF, 
BEARING,  G,  ALT., 
VELOCITIES,  CLOSURE , 
TRACKING  ACCURACY, 
LAUNCH  PARAMETERS, 
SET-UP.  . 
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recording  of  the  deviation  from  the  runway  centerline  during 
takeoff  roll):  (1)  parameters  for  implementing  logic  to  start 

and  stop  measurement  computations  (S) , (2)  information  related  to 
desired  performance  (D) , and  (3)  error  information  derived  from 
the  differences  between  actual  and  desired  performance  (E) . 


When  a list  of  parameters  was  developed  to  show  all  the 
required  sensors  and  recording  needed  for  total  measurement,  it 
was  seen  that  the  resulting  parameters  could  be  placed  in  the 
following  overlapping  categories:  (1)  pictorial  information 

(e.g.,  out-the-window  radar),  (2)  analog  information  (e.g.,  time- 
varying  quantitative,  such  as  airspeed),  (3)  discrete  information 
(e.g.,  weapon  release),  (4)  audio  information  (e.g.,  communica- 
tions) , and  (5)  desired  performance  and  existing  conditions. 

These  parameters  were  later  assigned  to  alternative  devices  for 
data  acquisition.  After  tradeoff  analyses  were  conducted,  a 
hybrid  audio-video/photo-digital  recording  system  was  adopted. 
Although  many  parameters  could  be  acquired  by  either  video-photo 
or  digital  recording  devices  (to  allow  a partial  system  to  have 
a stand-alone  capability) , tentative  parameter  allocations  are 
listed  in  Tables  4 and  5.  Audio  recording  will  be  accomplished 
with  either  video-photo  or  digital  recording  devices.  Desired 
performance  and  conditions  are  manually  derived  from  briefing/ 
debriefing  sessions  and  documents.  Additionally,  spatial 
coordinates  (X-Y  data)  may  be  obtained  if  data  are  collected  on 
an  instrumental  range  including  tracking-radar  equipment, 
although  equivalent  information  may  be  available  from  video-photo 
recording  as  shown  in  Table  6. 


After  detailed  trade-off  analyses,  the  use  of  video  or  photo 
techniques  was  emphasized  due  to  lower  costs,  flexibility  of 
application  and  simpler  development  compared  to  other  all- 
electronic  techniques;  however,  the  problems  associated  with 
cockpit  installation  and  an  unfortunate  tendency  for  loss  of  data 
should  be  noted,  but  it  is  believed  necessary  to  accept  these 
deficiencies  for  low-cost  combat-crew  training  application  (see 
Volume  VI). 


Accuracy  requirements  for  measurement  parameter  sensing  and 
recording  are  listed  at  the  right-hand  side  of  Table  4.  These 
accuracies  are  referenced  to  the  information  displayed  to  the 
crew  (for  example,  the  required  airspeed  tolerance  is  ± 1%  of  the 
deviation  between  recorded  values  and  those  displayed  to  the 
pilot  on  his  airspeed  indicator)  since  the  criterion  given  the 
crew  is  that  they  maintain  vehicle  parameters  within  specified 
tolerances  referenced  to  their  instruments  (typically  5-10  knots 
for  airspeed  control) . The  tolerances  listed  in  Table  4 are 
approximately  l/10th  the  tolerances  required  of  the  crew. 


Measurement  Descriptions 


The  gross  operations  involved  in  measurement  computation  are 
presented  in  the  flow  diagram  in  Figure  7.  Each  parameter  must 
be  sampled  (at  a sampling  rate  of  2,  10  or  20  times  a second. 
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TABLE  4 

PARAMETERS  FOR  DIGITAL  DATA  ACQUISITION 
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AIRCRAFT  PARAMETERS 


1.  Pitch  (Pitch  Rate) 

X 

X 

X X 

X 

X 

X 

+1  degree 

2.  Roll 

X 

X 

X X 

X 

X 

X 

±1  degree 

3.  Heading 

X 

X 

X 

X X 

X 

X 

X 

±1  degree 

4.  Airspeed 

X 

X 

X 

X X 

X 

X 

X 

±1  knot 

5.  MACH 

X 

X 

X 

±.02  MACH 

6.  Altitude 

X 

X 

XXX 

X 

X 

X 

±10  feet 

7.  Vert.  Vel. 

X 

X 

XXX 

X 

X 

X 

±50  fpm 

8.  Angle  of  Attack 

X 

X 

X 

X 

X 

±1  unit 

9.  Acceleration  (G) , 

X 

X 

X 

X 

±.5G 

10.  Power  (RPM,  EPR,  TIT,  Fuel 

Flow) 

X 

X 

XXX 

X 

X 

X 

±1%  Full  Scale 

11.  Fuel  Quantity 

XX 

X 

±5%  Full  Scale 

CONTROL  PARAMETERS 

1.  Stick  (Pitch) 

X 

X 

X 

±5%  Full  Scale 

2.  Stick  (Roll) 

X 

X 

X 

±5%  Full  Scale 

3.  Rudder 

X 

±5%  Full  Scale 

4.  Flap  Position 

X 

X 

X 

±5%  Full  Scale 

5.  Stab  Trim  Position 

X 

X 

X 

±5%  Full  Scale 

BINARY  DISCRETE  PARAMETERS 

1.  Thru3t  Reverse 

X 

— 

2 . Speed  Brakes 

X 

X 

X 

X 

— 

,4.  Main,  Nose  Gear  Contact 

X 

— 

5.  Nose  Steer  Engaged 

X 

— 

6.  Gear  Select 

X 

— 

7.  Drag  Chute 

X 

— 

8.  Wheel  Brakes 

X 

X 

-- 

10.  Red,  Green  Light 

X 

— 

11.  Weapon  Release  (Pickle). 

X 

X 

X 

X 

-- 

16.  Crewmember  Voice  Switch 

X 

X 

X 

XXX 

X 

X 

X 

— 

19.  Marker  Beacon 

X 

— 

24.  Event  Marker 

X 

X 

X 

XX  X 

X 

X 

X 

— — 

TIME 

1.  GMT  (Range  Time) 

X 

X 

X 

XX  X 

X 

X 

X 

Hrs,  Min,  Sec, 

1/100  Sec. 
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TABLE  5 

PARAMETERS  FOR  VIDEO/PHOTO  DATA  ACQUISITION 


PHASE 

PARAMETERS 

Transition 

Runway  Centerline  Deviation,  Lateral 
Drift,  Threshold  Crossing,  Distance  Down 
Runway,  Ground  Track. 

Instruments 

TACAN  '.Frequ.,  Course  Set,  Course  Error, 
Bearing,  DME) ; 

VOR  (Frequ.,  Course  Set,  Course  Error, 
Bearing) ; 

ILS  (Frequ.,  Localizer  Error,  Glide  Slope 
Error,  Marker  Beacon) . 

Intercept 

• 

Target  use  of  ECM,  Maneuvering 
Radar:  Azimuth,  Elevation,  Range,  Range 

Rate,  Range  Gate,  Steering  dot  error, 
Firing  Circle  Radius,  Lockon  Pulse,  IF 
Gain,  Video  Gain,  Erase  Intensity. 

Refueling 

Range  to  Tanker,  Range  Rate,  Probe 
engagement,  Centerline  Displacement, 
Lights  (Up,  Down,  Fore,  Aft),  Attitude 
Error . 

Air  Drop 

Cross track  Error,  Groundspeed,  Terrain 
Clearance,  Range/Bearing/ AAltitude  from 
Lead  Acrft,  Red/Green  Drop  Lights,  Actual 
Air  Release  Point. 

Formation 

Spacing:  Range,  Range  Rate,  Bearing, 

AAltitude. 

Ground  Attack 

Target  Slant  Range,  Aim  Point  Error,  Bomb 
Fall  Line,  Flight  Path  Error,  Spacing 
in  Range  Pattern. 

Dart  Firing 

Range,  Azimuth,  Elevation,  Hits. 

Air  Con  bat  fe-^rrers 

Target  Range,  Range  Rate,  Aspect  Angle, 
Heading  Crossing  Angle,  Elevation. 

1 


TABLE  6.  REQUIREMENTS  FOR  X-Y  DATA 


PHASE 

PARAMETER 

OBTAINABLE  WITH 
VIDEO/PHOTO  ? 

1 

TRANSITION 

GROUND  TRACK 

RA* 

2 

CENTERLINE  DEV 

RA 

3 

LAT.  DRIFT 

NO 

4 

THRESHOLD 

RA 

5 

DIST.  DOWN  RNWY 

RA 

6 

SPACING 

I \ 

7 

INTERCEPT 

TGT.  AZIMUTH 

I o 

8 

(PRIOR  TO  LOCKON) 

TGT.  ELEVATION 

NO 

9 

TGT.  RANGE 

NO  , 

10 

TGT.  RANGE  RATE 

No 

11 

TGT.  ASPECT  ANGLE 

No 

12 

AIR  REFUELING 

TANKER  RANGE 

RA 

13 

TANKER  RANGE  RATE 

RA 

14 

CENTERLINE  DISPL. 

Yes 

15 

LIGHTS  UP 

Yes** 

16 

DOWN 

Yes** 

17 

FORE 

Yes** 

18 

AFT 

Yes** 

19 

ALTITUDE  ERROR 

• RA 

20 

AIR  DROP 

CROSS  TRACK  ERROR 

No 

21 

POSITION  ERROR 

Yes 

22 

RANGE  FROM  LEAD 

RA 

23 

BEARING  FROM  a-RAD 

RA 

24 

A ALTITUDE  FROM  LEAD 

RA 

25 

ACTUAL  AIR  RELEASE  PT. 

Yv  j 

26 

FORMATION 

RANGE 

RA 

27 

RANGE  RATE 

RA 

28 

BEAR.TNs’3 

RA 

29 

GROUND  ATTACK 

TGT.  SUiNT  RANGE 

RA 

30 

AIM  POINT  ERROR 

Yes 

31 

BOMB  FALL  LINE 

Yes 

32 

FLIGHT  PATH 

RA 

33 

SPACING 

RA 

34 

DART  FIRING 

RANGE 

RA 

35 

AZIMUTH 

RA 

36 

ELEVATION 

Yes 

37 

AIR  COMBAT 

TGT.  RANGE 

RA 

38 

TGT.  RANGE  RATE 

RA 

39 

TGT.  ASPECT  ANGLE 

RA 

40 

TGT.  HDG  CROSS  ANGLE 

No 

41 

ELEVATION 

RA 

42 

SPACE  PATH 

NO 

*RA  = Reduced  Accuracy. 

•♦Obtainable  with  Video/Photo  System, 
but  not  easily  otherwise. 
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depending  on  the  application)  and  tested  to  determine  if  condi- 
tions are  appropriate  to  start  measurement  computation,  and 
later  to  stop  computation.  During  the  measurement  interval  (or 
at  specific  conditions,  e.g.,  f laps-up) , one  of  the  following 
statistics  is  calculated:  (1)  the  value  of  a parameter,  (2) 

minimum  value,  (3)  maximum  value,  (4)  mean,  or  (5)  standard 
deviation.  Thus,  measurement  is  defined  by  specifying  start/stop 
conditions  and  one  of  the  five  statistics. 


An  example  of  a measurement  specification  patterned  in  this 
fashion  is  shown  in  Figure  3.  Measurement  specifications  were 
produced  for  each  common  training  phase. 


While  these  specifications  initially  assume  automatic 
recording  and  computation,  they  can  also  be  used  to  describe 
manual  data  processing  procedures.  If  parameters  are  recorded 
which  clearly  identify  start/stop  conditions  for  automatic 
processing  (e.g.,  a discrete  signal  indicating  wheels-up) 1 . 

These  parameters  can  also  be  used  to  start  and  stop  manual 
processing  such  as  scanning  for  out-of-tolerance  conditions. 
During  the  measurement  interval,  the  value,  minimum  or  maximum  of 
a parameter  may  be  determined  manually,  but  manual  processing  for 
computation  of  a mean  or  standard  deviation  is  judged  to  be 
excessively  laborious  and  time-consuming  since  a large  number  of 
data  samples  (at  2 or  more  times  a second)  is  necessary. 
Consequently,  the  measurement  specifications,  as  exemplified  in 
Figure  3,  are  suitable  for  (1)  defining  software  for  digital 
computer  measurement  processing,  or  (2)  manual  processing 
procedures . 


Communication  Analysis 


\ 


Measurement  for  crew  performance  overlays  the  system 
performance  measurement  thus  far  discussed.  The  performance 
demonstrated  in  A- 7 or  F-106  aircraft  clearly  involves  only  one 
man;  however,  the  same  missions  may  be  flown  by  two  men  in  the 
F-4  aircraft,  requiring  additional  measurement  to  investigate 
crew  interaction  and  diagnose  individual  performance  contribu- 
tions. The  combat  airlift  mission  in  the  C-130  and  C-141 
aircraft  involves  such  close  coordination  of  pilots,  navigators, 
loadmasters  and  engineers  that  it  is  difficult  to  isolate  an 
individual's  performance  (even  the  crewmembers  themselves 
cannot  be  sure  of  the  adequacy  of  their  performance) . 


An  individual's  performance  can  be  assessed  by  relating 
measurement  at  his  workstation  to  overall  system  performance; 
this  type  of  measurement  is  subsumed  under  the  previous  measure- 
ment discussion.  However,  the  interaction  between  crewmembers 


1 Therefore , each  recording  device  potentially  requiring  manual 
processing  (e.g.,  video  recording)  should  permit  a means  for 
discrete  signal  recording. 
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requires  analysis  of  communications — presenting  somewhat 
different  requirements  than  system  performance  measurement.  The 
following  paragraphs  present  important  topics  related  to  crew 
performance  measurement  (see  also  Volume  IV). 

Communication  measurement  categories.  Communications  mea- 
surement must  treat  at  least  two  gross  types  of  crew  interaction: 

(1)  information  is  exchanged  to  aid  another  crewmember  in 
performing  his  duties  (e.g.,  when  the  F-4  Weapon  System  Officer 
acts  as  a good  "copilot")/  and  (2)  a crewmember  provides  a 
directive  role  in  guiding  another's  performance  (e.g.,  when  the 
F-4  Weapon  System  Officer  provides  directive  commentary  to  the 
Aircraft  Commander  in  air-air  intercept) . In  the  latter  case, 
direct  links  between  auditory  commands  and  system  performance  can 
be  identified,  allowing  communication  to  be  measured  in  terms  of 
resulting  performance  changes. 

Six  categories  of  measurement  related  to  information 
transfer  were  examined: 

(1)  Timing.  Measures  of  information  timing  should  relate 
to  (1)  jamming  more  important  messages,  (2)  providing  information 
at  the  wrong  time,  (3)  delay  in  providing  information,  and  (4) 
providing  information  at  a rate  not  permitting  effective  response 
by  another  crewmember. 

(2)  Accuracy.  Measures  of  accuracy  require  comparison  of 
what  is  said  in  relation  to  the  measured  situation  (e.g.,  was 
altitude  reported  correctly?) . 

(3)  Brevity.  As  radio  and  interphone  traffic  often  exceed 
channel  capacity  in  combat,  measurement  should  address  communica- 
tion duration  and  comparison  to  the  standard  vocabulary  of  the 
operational  brevity  code. 

(4)  Number  and  frequency.  Also  in  relation  to  communication 
brevity,  the  number  of  communications  and  frequency  of  communica- 
tions can  be  measured. 

(5)  Information  content.  Measures  of  information  trans- 
mitted per  unit  time  are  quite  important  although  usually 
practically  difficult  to  obtain.  Other  measures  such  as  time, 
number  and  frequency  are  often  confounded  since  a crewmember  may 
convey  much  information  in  short  time  or  few  transmissions,  while 
another  may  say  little  in  a long  time  or  many  transmissions; 
without  knowledge  of  the  information  content  it  would  be  diffi- 
cult to  evaluate  these  situations. 

(6)  Performance  changes.  The  performance  changes  of  the 
vehicle,  desired  as  a result  of  communicating,  define  measurement 
in  terms  of  links  between  auditory  data  and  system/mission 
performance  data.  For  example,  turning  performance  can  be 
measured  following  a "hard-as-possible"  direction  to  the  pilot  of 
an  F-4  aircraft. 
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Auditory  data  processing.  A computer-assisted  manual 
auditory  processing  system  is  required  since  automatic  voice 
decoding  equipment  is  not  available.  Expert  personnel  are 
therefore  required  to  identify  complex  performance  and  to 
structure  processing  rules  for  data  clerks  who  reduce  data  to  a 
form  allowing  input  to  a computer. 


Three  auditory  data  processing  problems  impose  requirements 
for  measurement  system  design: 

(1)  Identification.  In  spite  of  normal  communications 
clutter,  it  is  necessary  to  clearly  identify  who  is  talking, 
ever,  if  two  transmissions  are  simultaneously  made  (jamming). 
Voice  key  circuitry  is  recommended  to  provide  a digital  signal 
indicating  which  crewmember  is  talking. 

(2)  Synchronization.  Audio  data  must  be  synchronized  with 
other  data  recording  to  permit  relating  auditory  information  to 
corresponding  performance  changes.  Each  recording  device  must 
therefore  include  an  audio  voice  track. 


(3)  Data  reduction.  Manual  functions  in  audio  data 
processing  are  unavoidable,  but  can  be  minimized  through 
computer-assistance.  Audio  data  playback  must  be  accompanied  by 
display  of  performance  parameters,  especially  the  parameter  of 
TIME,  to  allow  computer  correlation  of  manual  audio-data  entry 
with  the  digital  data  base.  A convenient  means  for  manual  data 
entry  is  a desirable  feature. 

These  data  processing  problems  must  be  considered  in  the 
design  of  a combat-crew  training  measurement  system. 
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III.  THE  PERFORMANCE  MEASUREMENT  SYSTEM 


The  second  product  of  this  study  was  the  detailed  descrip- 
tion of  a combat-crew  performance  measurement  system.  The 
measurement  system  description  in  this  chapter  includes  (1)  the 
data  acquisition  subsystem,  (2)  the  data  processing  subsystem, 
(3)  the  personnel  subsystem,  and  (4)  facilities.  The  use  of 
off-the-shelf  equipment  and  proven  techniques  were  emphasized 
to  ensure  that  the  system  would  be  workable  and  attainable. 


Of  course,'  the  principal  objectives  in  measurement  system 
design  were  to  produce  needed  information,  by  available  devices 
and  techniques,  in  the  combat-crew  training  environment,  and  at 
low  cost.  A large  portion  of  the  design  tradeoff  analyses  were 
devoted  to  comparison  of  video  or  photographic  recording 
techniques  with  digital  recording  techniques.  The  ultimate 
conclusion  was  that  a hybrid  system  of  video,  photo,  and  digital 
devices  should  be  assembled  to  allow  use  of  the  best  features  of 
each;  however,  the  video/photo  and  digital  subsystems  were  to 
have  sufficient  capability  to  allow  separate  deployment. 


Separate  deployment  of  the  video/photo  and  di.gital  subsystem 
was  desired  since  either  subsystem  alone  could  suffice  for  a 
number  of  potential  applications  (i.e.,  for  simplified  measurement 
and  specific  combat-crew  training  phases).  Therefore,  a modular 
system  was  designed  to  allow  either  subsystem  to  be  used 
separately  where  this  is  suitable,  and  used  together  as  needed 
(in  particular,  bofh  subsystems  are  needed  where  detailed 
measurement  is  required  throughout  the  combat-crew  training 
phases).  Of  course,  the  modular  principle  also  permits  separate 
procurement  of  subsystems  at  different  times,  where  the  immediate 
applications  and  available  funding  dictate  this  choice. 


Even  though  a video/photo  system  would  require  a significant 
amount  of  manual  data  processing,  a general-purpose  computer  is 
required  to  accept  digital  and  manual  inputs,  edit  and  assemble 
data  files,  compute  measurement,  and  perform  research  analyses. 

As  data  collection  may  take  place  at  a number  of  training  sites, 
the  ground-based  portions  of  the  system  are  to  be  housed  in 
trailers  fashioned  afterLjnobile  homes.  These  facilities  will 
include  debriefing  features~~tor~interface  with  ongoing  training, 
as  well  as  a full  complement  of  persbimeX-j^apable  of  pursuing 
research  goals . 


System  Criteria 


As  the  measurement  system  is  at  least  to  be  a tool  for 
research  and  combat-crew  training,  the  system  criteria  dealt 
mainly  with  the  use  of  the  tool  and  relevant  constraints.  The 
criteria  were  developed  early,  before  any  design  considerations. 
Each  requirement  is  discussed  below  in  terms  of  tradeoffs 
considered  (definitions  and  general  discussion  appear  in 
Chapter  I) . 
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Provide  needed  information.  Based  on  the  investigations 
conducted,  needed  information  will  be  provided  if  (1)  necessary 
parameters  are  sensed,  (2)  sensors  are  available  at  each  source 
of  data,  and  (3)  meaningful  measures  can  be  computed  from  the 
raw  data. 


(1)  Sense  necessary  parameters.  Sensors  must  be  allocated 
to  audio,  pictorial,  analog,  discrete,  and  space-coordinate  data 
to  support  combat-crew  training  information  needs.  Audio  and 
space-coordinate  data  present  special  sensor  problems,  the 
remaining  parameters  are  allocated  fro  video/photo  and  digital 
data  acquisition  equipment.  Of  course,  some  information  is  also 
manually  acquired  and  manually  processed. 


Audio  recording  of  voice  communications  was  found  to  be  of 
value  in  all  phases  of  combat-crew  training.  Audio  recording  must 
be  synchronized  with  other  recorded  information;  therefore,  a 
voice  track  must  be  included  with  both  video  and  digital  recorders. 


Space-coordinate  recording  (X-Y-Z  data)  requires  the  use  of 
expensive  multiple- target  tracking  radar  for  complete  coverage 
with  high  accuracy.  An  analysis  (see  Table  6)  fortunately 
indicated  that  satisfactory  information  could  be  obtained  for 
many  envisioned  applications  with  video  or  photographic  recording. 
Therefore,  a tracking  radar  was  not  included  in  the  system  design, 
but  an  option  remains  to  accept  and  process  data  from  such  a 
facility.  Since  comprehensive  measurement  of  air  combat  ’ 
maneuvering  requires  information  on  the  space  paths  of  combating 
aircraft,  radar  tracking  data  will  be  required  for  detailed 
measurement;  one  is  otherwise  limited  to  position  information 
available  through  video/photo  cameras  only  when  the  other  aircraft 
is'  within  the  camera  fie  Id- of- view. 


Many  kinds  of  out- the -window  information  can  be  obtained 
with  video/photo  recording  which  would  otherwise  be  difficult  and 
expensive  to  acquire,  while  at  the  same  time  allowing  sampling  of 
information  from  the  cockpit  instrument  panel.  The  primary 
virtues  of  automatic  digital  recording  are:  (1)  high-speed  and 

accuracy,  (2)  ability  to  sense  information  which  cannot  be 
directly  seen  with  a camera  in  the  cockpit,  and  (3)  automated 
computer  processing  permitting  complex  sophisticated  measurement. 


It  was  concluded  that  a hybrid  system,  including  both  video/ 
photo  and  digital  recording,  is  needed  to  attempt  to  provide  a 
simple  system  sensing  all  combat-crew  training  measurement 
parameters;  however,  to  allow  flexibility  in  deployment,  the 
video/photo  and  digital  equipment  are  each  expected  to  provide  an 
independent  capability  for  limited  data  acquisition  (stand-alone 
capability) . Due  to  considerations  of  cost,  reliability  and 
timely  research  execution,  the  complexity  of  the  digital  record- 
ing equipment  was  restrained,  but  with  sufficient  overlap  with 
video/photo  data  acquisition  to  provide  a digital  stand-alone 
capability  and  enhance  reliability  of  data  collection  with  the 
hybrid  system.  On  the  other  hand,  discrete  recording  channels 


are  required  for  the  video/photo  recording  equipment  to  enhance 
the  video/photo  stand-alone  capability  and  permit  more  efficient 
manual  data  processing.  Figure  8 depicts  the  allocation  of  data 
acquisition  responsibility  to  video/photo  and  digital  recording 
devices . 


(2)  Sensors  at  each  source  of  data.  Combat-crew  training 
performance  information  may  come  from  a number  of  sources  and 
through  a variety  of  media.  The  data  acquisition  subsystem  must 
acquire  all  forms  of  information  and  interface  with  manual  and 
automatic  means  for  compiling  an  integrated,  correlated,  common 
raw  data  base.  Five  categories  of  data  sources  must  be  treated: 
(1)  aircraft  and  simulator,  (2)  field  (e.g.,  range  or  runway), 

(3)  briefing  and  debriefing,  (4)  external  sources  (e.g.,  other 
studies,  tracking  radar),  and  (5)  documents.  The  data  acquisi- 
tion subsystem  therefore  must  contain  all  the  components  of  the 
block  diagram  shown  in  Figure  9. 


(3)  Compute  meaningful  measures.  Although  the  raw  data 
may  be  interpretable  to  a degree  in  its  original  form  (such  as, 
viewing  a video  playback  monitor) , the  data  take  on  more  meaning 
when  calculations  are  made  using  the  raw  data  to  provide 
comparable  indices  of  performance.  The  hybrid  system  recom- 
mended requires  a combination  of  manual  and  digital  computer 
processing.  Therefore,  a manual  data  processing  workstation  and 
a general  purpose  digital  computer  are  included  in  the  recom- 
mended design. 


Useful  format.  The  superiority  of  formats  depends  on  the 
use  of  the  information.  The  video/photo  format  presents  informa- 
tion in  much  the  same  way  as  the  information  is  presented  in  * 
flight;  this  should  permit  ease  of  interpretation  by  the 
instructor  pilot  and  student,  forming  a framework  to  improve 
communication  between  each  other,  or  between  either  of  them  and 
the  research  scientists.  On  the  other  hand,  a quantitative 
presentation  of  information,  in  a much  different  form  than  that 
presented  in  the  cockpit,  promises  hope  for  greater  objectivity 
and  the  solution  of  problems  which  appear  vague  when  discussed 
in  terms  of  ordinary  flight  parameters.  Both  pictorial  and 
numerical/ graphical  formats  are  therefore  likely  to  be  desired, 
and  both  are  included  in  the  recommended  design. 


Research  cycle  time . Measurement  processing  (exclusive  of 
data  analysis)  consists ”of  the  following  time  periods:  (1)  air- 

craft or  simulator  modification,  installation,  and  checkout, 

(2)  equipment  calibration  and  maintenance,  (3)  data  collection, 

(4)  data  conversion  and  generation  of  quick- look  data,  and 
(5ti  measurement  computation. 


\ Equipment  modification,  installation  and  checkout  introduce 
a significant  time  delay  (Ti)  before  productive  data  collection 
can  begin.  Calibration  and  maintenance  time  (T2)  reduces  the 
availability  of  equipment  for  productive  use.  As  digital  record- 
ing equipment  has  historically  involved  much  time  for  both  Ti  and 
T2,  its  complexity  was  minimized. 
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Figure  9.  Data  Acquisition  Block  Diagram. 
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Data  collection  is  a critical  period.  It  is  probable  that 
a concerted  effort  will  be  required  to  process  measurement  in 
pace  with  a training  squadron  schedule.  Each  airborne  data 
collection  system  may  be  required  for  more  than  a flight  a day 
for  five  days  of  the  week.  Inherent  equipment  reliability  is 
nseded  to  avoid  large  amounts  of  maintenance  manpower,  and 
associated  recurring  costs. 

A form  of  quick-look  data  is  required  of  each  data  acquisi- 
tion device.  Video  recording  permits  rapid  playback,  but  digital 
recording  requires  some  form  of  conversion  and  digital  computer 
processing,  and  photo  processing  requires  ready  access  to  speedy 
developing  services. 

Measurement  computation  time  is  not  expected  to  be  signifi- 
cant for  digital  recording  once  needed  software  is  checked-out; 
however,  manual  processing  of  video/photo  and  audio  data  require 
efficient  techniques  and  procedures  (it  is  estimated  that  manual 
processing  of  30-minutes  of  video  recording  can  be  accomplished 
in  less  than  1%  hours) . 

Costs.  Both  initial  and  recurring  costs  must  be  considered 
in  system  design  tradeoffs.  Use  of  off-the-shelf  equipment 
reduces  initial  costs,  although  costs  of  modifications  for 
airborne  application  must  be  considered. 

A primary  tradeoff  consideration  is  the  relatively-high  cost 
of  development  and  installation  of  a digital  recording  system  in 
comparison  with  the  relatively-costly  and  time-consuming  manual 
data  processing  procedures  associated  with  video/photo  recording. 
The  hybrid  system  selected  provides  a compromise  moderate  cost 
by  (1)  reducing  the  need  for  developing  and  installing  a large 
digital  system,  and  (2)  permitting  a significant  amount  of 
automated  data  processing.  The  total  cost  of  the  hybrid  system 
is  expected  to  be  less  than  a system  based  entirely  on  digital  or 
video/photo  techniques  (although  neither  technique  can  provide 
all  needed  information  by  itself)  . Gross  estimates  of  costs  for 
an  all-digital  system  compared  to  an  all-video  system  indicate 
that  the  all-digital  system  is  5-10  times  more  costly.  The 
amount  of  manual  data  processing  for  an  a 11- video /photo  system 
is  a f nction  of  the  number  of-  data  collection  flights  per  day, 
but  is  always  undesirable  and  costly  in  comparison  to  semi- 
automatic processing  for  an  all-digital  system.  The  high  costs 
of  design,  development,  installation  and  test  associated  with  an 
all-digital  system  loomed  large  in  the  tradeoff  analyses,  leading 
to  a hybrid  design  minimizing  the  digital  recording  capability, 
yet  providing  full  data  coverage  for  the  total  system  and 
allowing  a digital  stand-alone  capability. 

It  should  also  be  noted  that,  while  video  methods  will  be 
normally  superior  to  photographic  methods,  total  costs  can  be 
reduced  by  using  photography,  and  that  the  lowest-cost  useful 
system  results  from  using  only  airborne  instrumentation  cameras 
with  an  audio  recorder. 
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Manpower  costs  are  the  primary  recurring  costs.  Design  for 
high  equipment  reliability  must  be  stressed  to  reduce  engineering 
and  technician  labor,  but  other  labor  categories  are  difficult  to 
uce  without  impairing  research- team  effectiveness.  The  man- 
oading  depends  on  the  manner  in  which  digital  and  video/photo 
techniques  are  used,  and  on  the  number  of  flights  to  be  flown 
for  data  collection  each  day.  For  four  or  more  flights  per  day, 
the  manloading  is  significantly  greater  for  an  all-digital  system 
assuming  equipment  reliability  demonstrated  by  current  flight 
test  efforts  (see  Volume  VI,  pp-  91-93)  . 


Data  distortion.  While  data  distortion  and  losses  are 
inevitable,  they  must  be  controlled  to  a minimum.  Loss  and 


distortion  of  data  have  been  common  in  previous  tests  of  video 
recording.  Tests  of  video  horizontal  and  vertical  resolution 
are  therefore  recommended  (it  is  estimated  that  resolution  must 
exceed  400  lines) . The  data  collected  from  any  device  must 
qlosely  compare  to  the  information  presented  to  the  crew  and  the 
errors  involved  must  be  small  compared  to  the  tolerances  which 
the  crewmember  is  expected  to  achieve  (note  accuracy  requirements 
in  Table  4) . 


Compatibility  with  training  devices.  Performance  measurement 
will  be  desired  in  the  simulator,  or  part-task  trainer,  as  well  as 
the  aircraft.  Video/photo  equipment  permits  easier  adaptation 


in 


to  new  installation  than  digital  recording  equipment  as  the  latter 
requires  expensive  cabling  and  sensors.  However,  there  is  merit 
in  using  one  system  for  all  applications  to  provide  a common 
output  for  efficient  data  processing,  and  thereby  producing 
identical  measurement  for  comparing  performance  between  training 
device  and  aircraft. 


Iterative  measurement  development.  Iterative  measurement 
development  requires  the  ability  to  change  measurement  as  a 
result  of  preliminary  measurement  tests.  Video  recording  is 
particularly  suited  to  trial-and-error  measurement  development 
since  (1)  desired  new  parameters  may  already  be  in  the  recorded 
field-of-view,  and  (2)  new  measures  are  obtainable  through  new 
instructions  to  data  clerks.  However,  even  with  video  data,  new 
measurement  transformations  may  be  computed  in  the  digital 
computer  from  the  raw  data  bank;  for  this,  flexible  software 
under  the  control  of  the  research  scientists  is  needed. 


Minimum  training  interference.  Any  measurement  device 
interferes  to  some  degree  with  the  process  being  measured,  but 
any  amount  is  undesirable.  Similarly,  the  measurement  system 
interference  with  training  must  be  minimiztl. 


It  is  possible  that  the  instructor  or  an  airborne  experi- 
menter may  be  able  to  start  and  stop  recording,  and  mark  special 
events  by  entering  a digital  code;  however,  the  recording 
sequence  should  be  automatically  controlled  as  much  as  is 
practical.  For  these  reasons,  programmed  sequences  of  on/off 
control  for  recorders,  and  convenient  manual  recording  control 
are  recommended. 
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External  data  correlation.  It  may  be  necessary  for  data 
from  a tracking  radar  station,  subjective  questionnaires, 
manuals,  or  other  experiments,  to  be  merged  with  video/photo  or 
digital  data.  Merging  of  data  will  be  performed  in  the  data 
processing  computer,  except  for  small  amounts  of  manual  merging. 
As  data  from  external  sources  may  occur  in  many  forms,  the 
computer  peripherals  should  include  most  standard  forms  of  data 
input  devices  (e.g.,  magnetic  tape,  punched  cards  and  paper  tape) 


Space,  weight,  cooling  and  power.  Data  acquisition  devices 
must  be  suitable  for  installation  in  at  least  the  sample  of 
aircraft  included  in  this  study  (i.e.,  B-52,  C-130,  C-141,  F-106, 
F-4,  A-7). 


Effective  personnel/facility.  The  personnel  of  the  research 
team  must  be  capable  of  performing  all  tasks  needed  to  achieve 
research  objectives,  including  (1)  planning,  (2)  test,  calibra- 
tion and  repair  of  equipment,  (3)  collection  and  processing  of 
data,  (4)  preparation  of  computer  programs,  (5)  interaction  with 
training  personnel,  (6)  improvement  of  measurement  and  research 
methods,  and  (7)  analysis  of  data  and  presentation  of  findings. 
The  facilities  and  equipment  must  support  these  activities. 

Since  research  may  successively  occur  at  different  and  remote 
sites,  mobile  trailers  are  recommended  to  house  personnel  and 
ground-based  equipment. 


Data  Acquisition  Subsystem 


The  Data  Acquisition  Subsystem  consists  of  the  equipment  (as 
shown  in  Figure  9)  for  data  recording  for  each  source  of  informa- 
tion and  devices  for  data  playback.  The  specifications  for  the 
required  equipment,  presented  in  Volume  VII,  are  summarized  in 
this  section. 


Aircraft/simulator  data  collection.  Although  a hybrid 
system  is  recommended,  it  is  also  required  that  the  video/photo 
and  digital  recording  systems  have  a stand-alone  capability; 
thus  three  data  acquisition  systems  are  defined:  (1)  a video/ 

photo  recording  system,  (2)  a digital  recording  system,  and 
(3)  a hybrid  recording  system.  Further,  an  option  is  available 
to  replace  video  equipment  with  airborne  photographic  cameras  if 
video  resolution,  size,  weight  and  costs  should  appear  unaccept- 
able upon  closer  examination  during  system  development. 


The  Aircraft/Simulator  Data  Collection  Station,  shown  in 
Figure  10,  consists  of  a dual-camera  Audio/Video  Recorder  (AVR) , 
an  Auxiliary  Camera  (AC),  an  Audio/Digital  Recorder  (ADR), 
Recorder  Controls/Displays  (RCD) , and  an  Interphone  Interface  (II) 


Most  pictorial  information  can  be  easily  obtained  only  with 
video/photo  techniques,  but  other  avionics  display  information 
(e.g.,  radar,  navigation  and  weapon  control)  will  also  be 
excluded  from  the  A*>R  to  reduce  its  complexity  and  associated 
costs.  Aircraft  and  control  parameters  (e.g.,  airspeed,  altitude) 
will  be  collected  on  the  ADR  and,  redundantly,  as  many  as  possible 
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within  the  field-of-view  of  the  AVR.  Similarly,  discrete  signals 
(e.g.,  wheels-up,  speed  brakes  out)  will  be  recorded  on  the  ADR, 
but  means  are  required  to  allow  recording  a small  number  of 
discrete  parameters  (e.g.,  weapons  release)  on  the  AVR  to  allow 
a stand-alone  capability  and  to  enhance  manual  measurement 
processing.  An  Auxiliary  Camera  (AC)  is  necessary  to  provide 
information  where  neither  the  ADR  nor  AVR  is  applicable  (e.g., 
during  air  drop,  pictures  directly  below  over  recognizable 
terrain  and  the  drop  zone) . Recording  Controls  and  Displays 
(RCD)  provide  manual  and  programmed  switching  of  recorders, 
information  for  equipment  set-up  and  for  onboard-experimenter 
functions.  Interface  with  the  interphone  system  (II)  couple tes 
the  aircraft/simulator  data  collection  station  which  is  recom- 
mended for  both  aircraft  and  simulator  environments. 


The  aircraft  modification  for  measurement  system  installa- 
tion must  be  consistent  with  space,  weight  and  power  availability, 
and  not  infringe  upon  aircrew  safety.  The  amount  of  aircraft 
modification  must  be  minimized.  The  installation  must  not  impair 
crew  vision  and  must  be  clear  of  seat  ejection  envelopes.  Good 
human  engineering  practices  and  maintainability  considerations 
are  necessary  for  efficient  inflight  operation  and  for  use  in  the 
environments  encountered  during  combat-crew  training. 


(1)  Audio/Video  Recorder  (AVR) . Two  video  cameras  are 
required  together  with  suitable  recording  equipment.  An  option 
for  connecting  a special-purpose  camera  instead  of  one  of  the 
standard  cameras  is  desired;  for  example,  a helmet-mounted 
camera  could  provide  unique  information  for  some  applications. 

Two  complete  images  should  be  recorded,  but  .in  lieu  of  this 
extensive  image  control  is  required  to  allow  freedom  in  combining 
the  camera  outputs  into  a composite  picture.  Audio  recording  of 
crew  communications  is  needed,  but  the  upper  frequencies  of  the 
available  bandwidth  may  be  used  for  recording  discrete  parameters 
as  audio-encoded  signals.  The  audio/video  recorder  must  permit 
remote  control  in  synchronization  with  other  recording  devices. 


Specifications  for  the  recording  and  playback  of  video 
imagery  are  summarized  in  Table  7 (more  detail  is  available  in 
Volume  VII).  In  particular,  the  principal  problems  noted  in 
previous  airborne  video  tests  must  be  removed. 


If  it  is  necessary  to  combine  the  output  of  two  video 
cameras  into  one  composite  picture  (instead  of  recording  two 
complete  pictures) , then  image  controls  are  desired  to:  (1)  split 

the  screen  horizontally,  one  camera  picture  in  the  upper  part, 
the  other  in  the  lower  part,  (2)  split  the  screen  vertically, 

(3)  inset  the  picture  from  one  camera  onto  the  other,  (4)  alter- 
nate pictures  at  periodic  intervals,  (5)  sequence  between  split 
screen  and  full  screen,  and  (6)  pan  one  camera  at  a selectable 
rate. 
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STANDARD  AMERICAN  TV 

VIDEO  QUALITY:  Horiz.  and  vert,  resolution,  500 

lines  desired,  400  lines  minimum. 

700-800%  contrast,  8-10  shades  of  gray. 

Greater  than  40  DB  signal- to-noise  ratio. 


TABLE  7 

AUDIO-VIDEO  RECORDING  AND  PLAYBACK  SPECIFICATIONS* 


] 


LENSES i (1)  Zoom  type,  (2)  Fixed-outside 

viewing,  (3)  Fixed-outside  viewing,  very  wide  angle, 

(4)  Fixed-cockpit  viewing. 


ILLUMINATION;  25-10,000  Foot-Candles ; auto-  j 

matic  sensitivity  control  over  this  range,  less  than 
approx.  .1-second  response  time,  rapid  recovery  from  •] 

direct  exposure  to  sun. 


AUDIO:  Frequency  response  100-10,000 

(min.)  HZ;  input  from  aircraft  interphone  system,  earphone 
& speaker  output. 


i 


! 


I 


! 


INTERFERENCE;  (Audio  & Video)  Low  noise  back- 

ground for  audio  communication  recording  (filters  for 
aircraft-induced  noise  required) , negligible  interference 
with  either  video  or  audio  signals  due  to  aircraft  radio 
transmission,  avionics,  or  weapons  firing. 

CONTROLS,  LIGHTS,  DISPLAYS:  Operation  with  either  hand  in 

aircraft  cockpit  with  full  flight  suit;  lights  adjustable 
for  day  and  night  operation;  no  restriction  to  crew 
visibility,  mobility,  or  safety;  amount  of  tape  remaining 
indicator  desired. 

SIZE,  WEIGHT,  POWER,  COOLING:  Suitable  for  fighter  aircraft 

installation;  without  significant  cockpit  modification, 
restriction  of  visibility,  mobility,  or  safety;  instal- 
lation clear  of  crew  ejection  envelopes. 

ENVIRONMENT ; Dictated  by  normal  combat-crew 

training  (e.g.,  altitude  to  50,000  ft.  MSL,  weapon  firing, 
turbulence,  maneuvering  + 8G) . 

MAINTAINABILITY ; Convenient  access  for  maintenance 

and  tape  handling;  provision  for  ground  operation  without 
aircraft  power;  set-up  and  test  equipment  to  be  included 
in  system,  including  operations  and  maintenance  documenta- 
tion. 

RECORDING  TIME;  30-minutes  minimum. 

REMOTE  CONTROL:  On/off,  s tart/s top  controlled 

remotely  in  unison  with  other  recording  devices. 


*The  difficulties  encountered  in  previous  tests  must  be  removed; 
i.e.,  specifically  those  in  (1)  TAC  Test  69-4F,  (2)  TAC-TR-70A- 
113F,  and  (3)  AFHRL (FT) -TRM-17 . 
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A method  must  be  developed  to  permit  recording  a minimum  of 
10  (24  desired)  discrete  parameters  within  the  AVR.  It  is  sug- 
gested  that  discrete  information  is  encoded  as  audio  tones  within 
the  available  bandwidth  above  3000  HERTZ.  A selectable  time  bit 
derived  from  a time  code  generator  must  be  recorded  as  one  of  the 
discrete  signals  to  aid  manual  data  processing  (entire  time  code 
is  recorded  on  the  ADR) . 

(2)  Audio/Digital  Recorder  (ADR)  . The  parameters  allocated 
to  audio/digital  recording  are  listedT  along  with  accuracy 
requirements,  in  Table  4 for  key  training  phases.  A minimum  of 
24  discrete  recording  channels  are  needed  in  addition  to  those 
required  for  digital  time  code  recording.  Of  course,  an  audio 
recording  channel  is  necessary.  Otherwise  there  are  16  channels 
which  require  analog-to-digital  conversion;  these  normally  must 
be  sampled  twice  a second,  but  for  some  measurement  a total  of 

10  channels  must  be  sampled  10  times  a second,  and  for  still 
other  cases,  a total  of  5 channels  must  be  sampled  20  times  a 
second.  It  is  important  to  note  that  the  accuracy  specified  for 
each  parameter  in  Table  4 refers  to  deviations  between  recorded 
information  and  that  indicated  to  the  pilot  by  cockpit  displays; 
otherwise,  data  collection  anomalies  may  be  attributed  to  pilot 
performance.  Table  8 summarizes  ADR  specifications  (more  detail 
is  available  in  Volume  VII). 

(3)  Auxiliary  Camera  (AC) . An  auxiliary  airborne  camera  is 
needed  as  a supplement  to  other  equipment  to  record  either 
displays  in  the  cockpit  or  external  views.  External  controls 
will  be  used  to  expose  the  film  in  either  a motion-picture 
sequence,  or  as  individual  frames. 
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(4)  Interphone  Interface  (II)  . The  interphone  interface 
must  permit  selection  of  communications  from  any  crewmember,  and 
generate  a discrete  signal  identifying  which  crewmember  is 
speaking.  Since  the  available  interphone  system  may  be  switched 
so  that  all  crewmembers  of  interest  are  not  recorded,  an  audio 
mixer  is  needed  to  combine  the  outputs  desired  for  both  AVR  and 
ADR  audio  recording.  Voice-operaited  switches  installed  at  each 
microphone  are  suggested  to  generate  a discrete  signal  whenever 
each  crewmember  speaks . 


(5)  Recording  Controls/Displays  (RCD) . Recording  controls 
and  displays  provide  manual  and  programmed  control  of  recorder 
functions,  and  information  for  set-up  and  data  collection.  The 
RCD  is  to  include  a time-code  generator  to  be  recorded  on 
discrete  channels  of  the  ADR  and  AVR.  Programmed  recording 
control  sequences  will  be  initiated  by  a manual  input  (an  event 
mark),  an  aircraft  discrete  parameter,  or  a specified  time  bit 
from  the  time-code  generator.  The  following  programmed  sequence 
is  desired:  (1)  recording  is  initiated  by  either  event  A or 

manually,  (2)  recording  is  stopped  manually,  at  event  B,  cr  after 
a specified  time,  (3)  recording  is  initiated  manually,  at  event 
C,  or  after  a specified  time,  and  (4)  recording  is  stopped  again 
manually,  at  event  D,  or  after  a specified  time. 
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TABLE  8 


AUDIO/DIGITAL  RECORDER  SPECIFICATIONS 


PARAMETERS /ACCURACY ; As  indicated  in  Table  4 (minimum)  , 

growth  capacity  desired. 


INTERFERENCE: 


Controlled  to  achieve  desired  accuracy, 


PARAMETER  SAMPLING  RATES:  Twice  per  second  (16  + discretes  + 

time),  10  times  per  second  (10  + discretes  + time),  20 
times  per  second  (5  + discretes  + time)  . 


AUDIO  RECORDING  CHANNEL:  Frequency  response  100  - 10,000  (Min) 

HZ  input  from  aircraft  interphone  system;  earphone  & 
speaker  output;  negligible  background  noise,  low 
distortion. 


TIME  CODE  RECORDING: 


(Hrs,  Min,  Sec,  1/100  Sec.) 


OUTPUT : Through  conversion  equipment  interface 

to  genera.',  purpose  digital  computer  . 

Audio  output  and  digital  display  of  digital  time  and 
discrete  parameters  during  playback. 


REMOTE  CONTROL:  On/Off,  Start-Stop,  together  with 

other  recording  devices. 


RECORDING  TIME: 


30-minutes  minimum. 


CONTROLS/LIGHTS/DISPLAYS : Operation  with  either  hand  in  air- 

craft with  full  flight  suit;  lights  adjustable  for  day  and 
night  operation;  no  restriction  to  crew  visibility, 
mobility,  or  safety. 


SIZE/ 


reiGHT/POWER/COOLING : Suitable  for  fighter  aircraft  instal- 

lation without  significant  cockpit  modification,  restriction 
of  mobility,  visibility  or  safety  (installation  clear  of 
ejection  envelope) . 


ENVIRONMENT:  Environment  dictated  by  normal  combat- 

crew  training  (e.g.,  altitude  to  50,000  ft.  MSL,  weapon 
firing,  maneuvering  ±8G) . 


MAINTAINABILITY : Convenient  access  for  maintenance, 

calibration  and  tape  handling;  provision  for  ground 
operation  without  aircraft  power;  test  stand  to  be 
provided  (other  than  standard  test  equipment)  ; minimum 
maintenance  personnel  requirements. 
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(6)  Test  Stand.  Since  maintenance  ease  with  minimal 
personnel  is  so  extremely  important,  the  data  acquisition  sub- 
system must  include  a test  stand  including  all  equipment  to 
permit  checkout  of  recorder  equipment  and  calibration  of  all 
channels  of  information  prior  to  each  data  collection  flight; 
the  same  equipment  should  facilitate  diagnosis  and  repair. 

Other  data  collection.  Data  will  be  collected  in  the  field 
at  remote  places,  at  briefings  and  debriefings,  from  documents, 
and  from  external  sources.  Each  of  these  must  be  considered  in 
addition  to  aircraft/simulator  data  collection  requirements. 

(1)  Field  data  collection.  Simple  portable  equipment  is 

required  for  data  collection  at  remote  sites  such  as  at  the 
runway,  the  weapons  range,  or  a siirveillance  radar  site.  The 
major  equipment  items  recommended  are:  (1)  a camera  with  inset 

clock  and  an  intervalometer  for  timed  exposure,  (2)  a small 
hand-carried  cassette- type  audio  recorder,  and  (3)  transceivers 
for  communicating  with  the  data  processing  facility  and  for 
monitoring  aircraft  transmissions. 

(2)  Briefing/debriefing  data  collection.  Much  of  the 
information  presented  during  mission  briefings  relates  to 
expected  performance  and  appropriate  performance  measurement; 
information  produced  during  debriefing  can  modify  the  briefed 
data,  and  can  prove  a source  of  subjective  measurement  for 
correlation  with  measured  performance  indicators.  A small  hand- 
carried  ca3sette-type  recorder  may  be  necessary  for  recording 
for  later  transcription.  A small  camera  may  also  be  useful. 

(3)  Documentary  data  collection.  Desired  performance  and 
measurement  control  parameters  will  also  be  obtained  from 
document:*  such  as  the  Dash-One  Technical  Order,  Phase  Manuals, 
and  operation  publications.  No  special  equipment  is  believed 
necessary  for  this  form  of  data  collection,  although  manual 
digital  entry  equipment  for  computer  processing  is  needed. 

(4)  External  data  collection.  Data  may  also  be  produced 
by  an  external  source  which  must  be  correlated  with  data 
collected  by  the  data  acquisition  subsystem.  However,  under  the 
assumption  that  these  data  must  be  provided  in  a form  permitting 
processing  with  the  available  general  purpose  digital  computer > 
no  special  equipment  is  needed  for  externally  produced  data. 

Data  playback.  The  data  playback  station,  as  shown  in 
Figure  11,  must  permit  the  transformation  of  data  collected 
through  the  five  avenues  shown  in  Figure  9 into  a digital  format 
appropriate  to  the  general  purpose  computer.  The  digital 
magnetic  tape  requires  only  routine  human  operator  activities  to 
load  the  data  into  computer  storage,  although  the"  audio  recording 
portion  is  necessarily  a manual  processing  task.  All  other  types 
of  information  require  manual  processing,  and  eventually,  typing 
of  formatted  data  onto  a punched  paper  tape. 
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(1)  Video  data  processing.  Video  information' will  be 
reproduced  through  two  video  monitors  if  dual-channel  recording 
is  possible,  otherwise,  one  will  be  used  if  split-screen  merging 
of  pictures  is  performed.  Communications  and  the  special  discrete 
audio  channels  are  reproduced  simultaneously  with  the  video 
information,  thus  the  operator  can  ascertain  who  is  talking,  and 
the  status  of  gear,  speed  brakes,  etc.  (if  these  are  chosen  for 
recording)  while  analyzing  the  video  content. 

The  discrete  information  is  also  useful  to  aid  searching 
for  key  events  to  initiate  measurement  activity  sequences. 
Automatic  measurement  processing  keys  on  clearly  identifiable 
events  whenever  possible  (i.e.,  discrete  events).  Unfortunately, 
these  events  often  do  not  appear  in  the  video  picture.  The  human 
operator  would  then  have  to  laboriously  search  the  video  tapes  to 
infer  from  the  cockpit  instruments  that  an  event  has  occurred 
which  is  important  to  measurement.  The  special  discrete  channels 
then  provide  the  human  operator  with  the  advantage  of  the  same 
information  provided  an  automatic  processor.  Since  it  is 
normally  desired  that  the  situation  at  or  near  each  event  be 
analyzed,  playback  control  features  are  required  to  stop  the 
video  tape  at  events  occurring  on  specified  discrete  channels. 

(For  example,  set  playback  controls  so  that  the  tape  will  stop 
when  weight  is  off  the  wheels,  allowing  conditions  at  lift-off  to 
be  recorded  for  computer  entry.) 

To  further  aid  the  human  operator  in  sampling  performance 
at  constant  time  intervals,  or  just  to  advance  the  tape  a known 
amount,  it  is  required  that  the  tape  be  advanced  on  command  for  a 
specified  number  of  seconds.  It  was  earlier  specified  that  a 
time  bit  (1,  2,  or  4 seconds)  be  recorded  on  one  discrete  channel 
so  that  these  bits  can  be  counted  on  playback  for  time  control 
(i.e.,  the  control  requested  is  to  advance  N events  on  the  time- 
bit  discrete  channel) . 

(2)  Digital  data  entry.  Digital  data  are  expected  to  come 
from  either  external  sources  or  the  audio/digital  recorder  system. 

External  digital  data  should  be  acceptable  in  magnetic  tape, 
punched  card,  or  paper-tape  form;  any  of  these  are  directly  read 
by  the  general  purpose  computer,  althougn  a magnetic  tape  copy 
may  be  made  for  high-speed  re-reading. 

Audio/digital  recording  can  result  in  two  acceptable  forms; 
a digital  magnetic  tape  which  can  be  converted  into  a magnetic 
tape  readable  on  standard  computer  magnetic  tape  units,  and  a 
digital  magnetic  tape  which  is  read  through  special  conversion 
equipment  directly  through  an  electrical  interface  into  the 
general-purpose  computer.  The  most  cost-effective  of  these 
alternatives  should  be  used.  Neither  method  requires  any 
significant  manual  processing  activities  to  enter  data  into  the 
computer  data  base. 


(1) 


Audio,  data  processing, 
sarily  a manual  task.  The  di£fe 


Audio  data  processing  is  neces- 
erence  between  audio  data  proces- 
sing with  audio/video  playback  and  the  audio/digital  playback  is 
the  method  of  correlating  the  audio  information  with  other 
performance  parameters.  With  audio/v.tdeo  playback,  the  magnetic 
tape  must  be  stopped  at  specific  auditory  events  (e.g.,  commands), 
selected  performance  parameter  values  noted,  and,  auditory  code 
and  values  punched  on  paper  tape.  With  audio/digital  playback, 
the  magnetic  tape  must  be  stopped  at  specific  auditory  events, 
the  time  noted  from  a display  of  the  digital  time  code,  and, 
auditory  code  and  time  punched  on  paper  tape  so  that  the  auditory 
information  can  be  correlated  with  appropriate  data  samples  read 
from  the  digital  magnetic  tape. 


(4)  Photo  data  processing.  Photo  data  processing  requires 
many  of  the  same  procedures  as  video  data  processing,  although 
time-lapse  photography  is  likely  to  be  the  primary  mode.  The 
resulting  information  must  be  coded  onto  punched  paper  tape, 
normally  including  time  values  to  allow  correlation  with  data 
from  other  sources. 


Data  Processing  Subsystem 


Processing  of  the  training  research  measurement  developed  in 
this  study  requires  appropriate  hardware  and  software.  The 
characteristics  of  these  must  be  discussed  to  aid  in  equipment/ 
software  selection  and  specify  new  software  development. 


Data  processing  hardware.  To  support  the  volume  of  data 
anticipated  it  is  necessary  to  have  all  measurement  processing 
tools  in  a central  location  ready  for  use  when  they  are  needed. 
The  equipment  required  in  the  dedicated  processing  facility  is 
shown  in  Figure  12 r 


The  heart  of  the  measurement  data  processing  facility  is  a 
general  purpose  digital  computer  (memory  16-32K  words,  word  size 
18-24  bit,  cycle  time  1-3  microseconds) . However,  much  of  the 
utility  of  the  system  for  measurement  and  data  analysis  depends 
on  the  presence  of  the  following  peripheral  equipment:  (1)  two 

magnetic  tape  units  (7-9  track,  variable  density) , (2)  disc 
units  (204  logical  units) , (3)  line  printer/plotter  (minimum  300 
lines  per  minute),  (4)  Teletype,  (5)  cathode-ray  tube  computer 
terminal,  (6)  paper  tape  reader  and  punch,  and  (7)  card  reader. 


If  both  the  data  format  and  physical  size  of  airborne 
magnetic  recordings  are  the  same  as  the  magnetic  tapes  hormally 
'produced  by  the  general  purpose  digital  computer,  then  only  the 
computer  magnetic  tape  units  will  be  needed  to  process  them. 
However,  since  this  is  not  likely,  conversion  equipment  will  be 
required  to  read  the  airborne  magnetic  tape  and  produce  and  input 
to  the  digital  computer. 


Software.  Data  processing  system  software  requirements  fall 
into  two  major  .categories:  (1)  the  executive  monitor,  and  (2) 

measurement  processing. 
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Figure  12.  Dedicated  Measurement  Processing  Facility. 


(1)  Executive  Monitor.  The  Executive  Monitor  supervises 

all  input/output  and  initiates  all  programs.  In  addition  to  its 
own  operating  system  and  housekeeping  routines,  the  monitor 
should  call  at  least  seven  utility  programs:  (1)  the  text  editor 

is  used  to  create  or  change  files,  (2)  the  file  manipulator  is 
used  to  perform  file  operations  such  as  transfer,  rename,  segment, 
combine,  delete,  verify,  and  copy,  (3)  the  FORTRAN  IV  compiler  is 
used  for  the  generation  of  most  performance  measurement  programs, 
(4)  the  machine  language  assembler  provides  a machine-language 
capability  for  user  programs,  FORTRAN-callable  subroutines,  and 
special-purpose  I/O  operations,  (5)  the  loader  is  commonly  re- 
quired to  enter  binary  object  programs  into  computer  memory  and 
start  their  execution,  (6)  the  debug  supervisor  is  used  for  the 
diagnosis  and  correction  of  new  programs  while  they  are  executed, 
and  (7)  chain  and  execution  features  permit  segmentation  and 
execution  of  programs  which  would  be  otherwise  too  large  to 
reside  in  computer  core  memory. 

(2)  Measurement  processing  software.  As  shown  in  Figure  13 
there  are  five  stages  of  performance  measurement  processing: 

(1)  ACQUIRE  DATA,  (2)  INPUT,  (3)  EDIT,  (4)  MEASURE  TRANSFORMATION, 
and  (5)  ANALYSIS.  Each  stage  requires  input  control  information 
and  performance  data  (left-hand  side  of  Figure  13) , and  produces 
needed  output  (right-hand  side  of  Figure  13). 


The  ACQUIRE  DATA  stage  reads  paper  tape  into  computer- 
controlled  files  (such  as  a test,  auditory  and  video/photo  data) 
and  allows  preliminary  editing.  A label  file  is  created  through 
the  keyboard.  The  system  test  editor  is  used  for  this  purpose. 

INPUT  arranges  and  labels  the  file  records  in  standard  form, 
scales  and  calibrates  data,  extracts  magnetic  tape  control 
information  from  all  files,  and  reads  the  external  magnetic  tapes 
into  a file.  All  data  files  which  will  be  subjected  to  numerical 
analysis  will  be  formatted  in  a standard  manner  consisting  of: 

(1)  a label  identifying  the  data  and  indicating  the  number  of 
words  of  data  to  follow,  and  (2)  a variable  size  data  array. 

Raw  data  files  are  tested  for  error  in  EDIT,  where  final 
corrections  are  made  and  data  are  placed  into  the  data  bank. 

EDIT  should  include  the  capability  to  correct  (1)  numeric  data 
within  a record,  (2)  labeling  data  within  the  label  record, 

(3)  record  manipulations  such  as  combining,  splitting  and  delet- 
ing  records  from  the  file.  EDIT  is  a man-computer  interactive 
program  which  requires  the  operator  to  make  data  corrections 
on-line  with  the  computer.  For  example,  errors  will  be  auto- 
matically flagged  according  to  specified  criteria,  then  upon  a 
keyboard  command  one  of  the  following  options  will  be  performed: 
(1)  replace  the  number  with  the  average  of  the  preceding  and 
following  numbers,  (2)  replace  the  number  with  a linear  fit  of 
the  preceding  two  numbers  or  the  following  two  numbers,  (3)  type 
in  a number,  (4)  delete  the  datum,  or  (5)  delete  the  error  flag 
indicating  the  datum  is  correct. 
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Following  operator  directions , measures  are  calculated  in 
MEASURE  TRANSFORM  and  placed  into  a temporary  analysis  file. 

Raw  data  is  read  from  the  data  bank  and  converted  into  specific 
measures  based  on  instructions  contained  in  a measurement  request 
file.  The  research  scientists  will  formulate  the  measurement 
request  file  for  trial  data,  receive  preliminary  output,  and 
iteratively  change  the  file  until  tho  desired  transformations 
evolve. 


ANALYSIS  routines  read  the  measures  and  perform  operator- 
directed  analyses.  ANALYSIS  is  a library  of  statistical  programs 
specially  tailored  to  the  research  requirements;  these  are 
general  purpose  programs  extensively  modified  for  easy  use  in 
training  research.  (e.g.,  include  tests  of  f-ratios,  clear 
labeling  of  results,  etc.) 


Hard  copy  is  available  at  each  of  the  five  stages  of 
processing. 


Personnel  Subsystem 


It  is  possible  to  identify  six  skill  types  needed  for 
effective  operation  of  the  performance  measurement  system;  (1)  a 
system  director,  (2)  research  personnel,  (3)  a computer  program- 


mer, (4)  data  clerks,  (5)  engineers  and  technicians,  and  (6)  a 


secretary.  A general  job  description  can  be  given  for  each  of 
these  skill  classes. 


System  director.  One  individual  must  be  responsible  foi. 
management  and  supervision  of  the  operating  system.  As  the 
success  of  a system  of  this  type  depends  to  a very  large  extent 
upon  careful  planning  and  continuous  control,  much  depends  upon 
this  individual.  All  levels  of  manloading  contemplated  require 
that  this  individual  also  serve  as  lead  experimenter  for  data 
collection  from  one  flight  per  day.  Among  other  skills,  this 
individual  is  expected  to  have  knowledge  and  experience  concerning 

(1)  combat  mission  flying  for  all  appropriate  aircraft  types, 

(2)  military  flight  training  procedures,  (3)  applied  experimental 
design,  (4)  computer  data  processing,  and  (5)  data  analysis, 
interpretation,  and  report  preparation. 


Research  personnel.  Trained  scientific  personnel  are 
necessary  to  support  all  phases  of  the  performance  measurement 
processing  ranging  from  experimental  planning,  conduct  of 
experimental  studies,  to  final  report  preparation.  During  the 
actual  conduct  of  experimental  studies,  the  research  specialist 
will  be  present  at  mission  briefings  to  extract  necessary 
information  relevant  to  performance  measurement.  In  some  cases, 
he  will  be  expected  to  serve  as  an  inflight  experimenter.  The 
research  specialist  will  be  responsible  for  supervision  of  data 
processing  for  his  assigned  flights,  and  may  perform  measurement 
debriefings  for  instructor  and  student  pilots.  The  research 
specialist  is  expected  to  be  a graduate  behavioral  scientist  with 
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knowledge  in  depth  of  military  missions  and  flight  training 
as  well  as  applied  behavioral  experimentation. 

Computer  programmer.  The  data  processing  subsystem  recom- 
mended  provides  generalized  software  with  the  assumption  that 
unique  programming  for  each  individual  study  will  be  required. 

This  approach  offers  the  greatest  flexibility  and  overall 
effectiveness  for  performance  measurement.  It  requires,  however, 
the  presence  of  a computer  programmer  both  to  set  the  specific 
study  software  and  to  insure  that  the  data  processing  programs 
are  running  properly.  For  inflight  and  simulator  experimentation, 
this  skill  type  has  consistently  proven  to  be  an  essential  member 
of  the  experimental  team. 

Data  clerk.  The  data  acquisition  subsystem  provides  a 
heterogeneous  set  of  input  data  which  must  be  processed  by  data 
clerks.  Personnel  tasks  are  required  for  manual  data  processing 
of  video,  digital,  audio,  and  photo  raw  data.  With  video,  photo 
or  audio  data,  the  end  product  from  the  data  clerks  is  a punched 
paper  tape.  In  the  actual  procedure,  two  types  of  data  clerks 
are  assumed:  (1)  clerks  for  video,  audio,  and  photo  raw  data 

extraction  and  (2)  teletype  operators  to  prepare  the  final 
punched  tape. 

Engineers  and  technicians.  It  is  possible  to  identify  ‘three 
types  of  engineers  and  technicians  necessary  for  equipment 
operation  and  maintenance  tasks:  (1)  checkout,  calibration,  and 

maintenance  of  airborne  (or  simulator)  data  acquisition  equipment, 
(2)  operation  and  maintenance  of  ground  data  processing  equipment, 
and  (-3)  utilization  of  photo,  video,  and  audio  equipment  in  the 
mobile  ground  facilities. 

Secretary.  A secretary  will  be  required  for  a myriad  of 
support  tasks  such  as  scheduling,  report  typing,  and  assisting  in 
data  collection  and  processing  when  needed. 

Estimated  personnel  requirements.  Table  9 presents  a summary 
of  the  estimated  personnel  requirements  for  the  performance 
measurement  system  as  a function  of  (1)  skill  types,  (2)  number 
of  flights  per  day  from  one  to  four,  and  (3)  two  levels  of  data 
acquisition  subsystem  complexity.  The  full  audio-video-photo- 
digital  hybrid  recording  system  has  been  assumed  here  as  one 
case,  and  is  termed  the  "complete  system;"  for  purposes  of 
comparison,  a "minimal  system"  consisting  of  a single  audio/video 
recording  system,  or  a single  cockpit  camera  plus  audio  recording, 
is  also  included. 

These  estimates  only  apply  to  the  inflight  experimentation 
case.  A somewhat  different  pattern  for  personnel  requirements 
develops  for  flight  simulator  performance  measurement,  and  total 
personnel  requirements  are  reduced  as  the  number  of  flights  per 
day  increases  to  four.  However,  the  availability  of  only  one 
simulator  is.  assumed,  eliminating  the  possibility  of  simultaneous 
flights,  but  creating  the  same  output  for  data  processing. 
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TABLE  9 

ESTIMATED  PERSONNEL  REQUIREMENTS 


SKILL 

CATEGORY 

System  Director 
Research  Personnel 
Programmer 
Data  Clerks 

(1)  Data  Transcription 

(2)  TTY  Tape  Punch 
Engineer/Technician 

(1)  A/B  Engineer 
A/B  Technician 

(2)  Comp.  Technician 

(3)  Equip:.  Technician 
Secretary 


COMPLETE  SYSTEM:  MINIMAL  SYSTEM: 

FLIGHTS/DAY  FLIGHTS/DAY 


1 2 3 4 1 2 3 4 


1 1111 
3 ’ 0 


1 1 1 

0 1 2 

0 0 0 

1 12 

1 1 1 

1 1 2 

1 2 3 

1 1 1 

1 1 2 

1 1 2 


1 0 

2 1 

2 0 

2 0 

4 1 

1 1 

2 1 

2 1 


1 2 2 
0 0 1 

11  1 
0 1 1 

0 0 0 
1 2 2 
1 11 
1 1 1 
11  1 


15  19 


5 6 10  11 


Facilities 

As  the  performance  measurement  system  undoubtedly  would  be 
required  to  function  at  more  than  one  combat-crew  training  site, 
and  since  the  time  for  equipment  set-up  involves  high  costs  and 
zero  productivity  in  terras  of  research  objectives,  it  is  recom- 
mended that  the  performance  measurement  system  be  housed  in  a 
mobile  facility.  The  mobile  facility  would  be  required  to  house: 
(1)  post-flight  debriefing  rooms,  (2)  data  processing  equipment, 
(3)  office  space  for  technical  personnel,  (4)  work  area  for 
equipment  technicians,  (5)  work  area  for  supporting  personnel; 

(6)  storage  area  for  records  and  data,  and  (7)  restroom 
facilities. 

These  facilities  could  be  mounted  in  several  truck-drawn 
trailers  or  in  one  or  two  large  mobile-home  trailers  devoid  of 
interior  partitions  and  furnishings  (with  the  exception  of 
restroom  facilities) . The  mobile  home  trailer,  with  reinforced 
floor,  is  recommended  for  flexibility  of  equipment  configurations, 
personnel  organization,  and  functional  utilization  of  available 
space . 

These  facilities  can  be  housed  in  either  one  12-foot  wide  by 
60-foot  long  mobile  home  type  of  vehicle,  or,  nore  advisedly,  in 
two  such  vehicles.  Figure  5 suggests  a possible  configuration 
for  two  vehicles;  the  one-trailer  layout  severely  limits 
debriefing/  office,  storage  and  work  space. 
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iy.  FUTURE  DEVELOPMENT  OPPORTUNITIES 


Opening  a new  flow  of  information  can  have  a major  influence 
on  training  technology.  The  execution  of  effective  contemporary 


training,  the  development  of  new  training  devices,  and  the 
exploitation  of  powerful  concepts  such  as  adaptive  training  and 
learner-centered  instruction,  all  depend  on  information  available 
through  performance  measurement.  Better  performance  information 
is  also  needed  to  realize  the  benefits  of  investments  already 
made  in  other  areas;  for  example,  the  fidelity  of  flight 
simulators  has  surged  in  the  last  decade,  but  performance 
measurement  to  convert  the  simulator  into  an  effective  training 
tool  has  not  made  corresponding  advances.  The  role  of  performance 
measurement  may  be  that  of  a causal  input,  a catalyst,  or  a weak 
link  in  a system  chain,  but  the  net  effect  of  better  performance 
measurement  in  any  case  is  a positive  and  possibly  revolutionary 
improvement . 


What  stands  in  the  way  of . improvements  through  performance 
measurement?  The  major  difficulty  at  present  is  that  adequate 
performance  measurement  systems  do  >not  exist  where  needed.  The 
present  study  has  defined  a blueprint  for  measurement  in  a manner 
permitting  adaptation  to  specific  needs  and  budgets.  The 
opportunity  for  future  performance  measurement  is,  therefore, 
available.  Recent  developments  indicate  a change  in  the  proper 
direction.  For  example,  the  Air  Force  is  developing  an  Advanced 
Simulator  for  Undergraduate  Pilot  Training  (ASUPT) , and  an  Air. 
Combat  Maneuvering  Range  (ACMR) . Both  permit  measurement  on  a 
large  scale. 


The  status  of  combat-crew  measurement  and  other  areas  of 
complex  man-machine  performance  is  reflected  by  the  measurement 
analysis  presented  in  this  report  (Chapter  II) . The  measurement 
described  here  reflects  the  evident  and  critical  dimensions  of 
performance,  and,  corresponding  to  the  initial  study  strategy, 
agrees  with  the  measurement  structure  used  by  operational 
training  personnel.  Given  this  measurement  capability  a number 
of  future  advances  are  possible.  First,  use  of  this  measurement 
will  lead  to  efficient  interim  and  immediately  available 
measurement  for  training.  Second,  improvements  in  the  generation 
of  optimally  efficient  measurement  sets  may  be  expected.  Thx^ 
clarification  of  the  relationship  between  objective  and 
subjective  measurement  should  be  possible.  None  of  these  results 
is  going  to  occur  without  an  acceptable  performance  measurement 
system  and  appropriate  experimentation. 


A number  of  avenues  can  be  taken  to  the  implementation  of  . 
the  measurement  system  described  in  this  report.  An  especially 
effective  and  thorough  method  is  shown  in  Figure  14  based  on  the 
methodological  steps  used  for  the  acquisition  of  large  weapon 
systems  (AFSCM  375-5) . This  system  management  methodology  is  the 
result  of  extensive  experience,  and  is  generally  as  applicable 
to  'small  systems  as  to  large  ones  (in  this  case,  scaled-down  as 
appropriate) . 
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Figure  14.  Measurement  System  Implementation  Plan 
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Figure  14  - (continued). 


It  is  assumed  that  a system  integration  contractor  is 
available  to  control  system  acquisition  under  the  direction  of 
the  Air  Force.  The  results  of  the  present  study  may  be  used  as 
the  essential  inputs  for  the  development  contract  (Figure  14 , 

Step  1.1).  After  selection  of  the  system  integration  contractor, 
final  detailed  design  is  accomplished  prior  to  hardware  and 
software  selection.  The  machinery  exists,  for  example,  to 
select  tradeoff  alternatives  such  as  the  most  effective  of  the 
off-the-shelf  video  recording  equipment.  Large  expenditures  for 
hardware  procurements  can  be  avoided  until  critical  design, 
analysis,  and  test  information  is  available.  A specific 
adaptation  of  the  normal  Category  II  tests  can  be  achieved  so 
that  the  tests  can  be  performed  while  collecting  research  data 
in  the  combat-crew  training  environment  and  while  personnel  are 
trained  to  carry  out  the  designated  research  mission. 

A schedule  for  the  acquisition,  test,  and  development  of  a 
specific  performance  measurement  system  is  presented  in  Figure 
15.  To  exploit  fully  the  benefits  of  the  developmental  framework 
recommended,  a 28-month  period  is  judged  to  be  necessary. 
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